CoCo Seminar - The Digital Education Revolution: initial data analysis of teacher and student reported use of laptops in year 10 science

After the publication of my first paper, I was humbled to be asked to present my work as part of the CoCo Seminar Series at the University of Sydney. My seminar covered the background of the Australian Digital Education Revolution, how it was implemented by the schools I work with, the methodology and results of my first paper, the methodology and results of my second paper (in press) and my current research. Below is the prezi from the presentation and below that is an explanation of each 'slide'. Any feedback or discussion would be greatly appreciated. You can also listen to and watch the actual webcast of the seminar. 

Slide 1 - my ever present microscope graphic (courtesy of Balázs Turai) with tinyurl link in homage to the diggers of Kokoda on the eve of ANZAC day (CoCoDER was already taken!).

Slide 2 - 7.30 Report from 4th Dec 2007 featuring Kevin Rudd, Mark Pesce and principals discussing the DER (post-election). Nice quote from Pesce that providing every student with a device will force the issue with teachers, policy makers and curriculum design.

Slide 3 - A Digital Education Revolution policy document (Rudd, Smith & Conroy, 2007 (pre-election)

Slide 4 - Melbourne Declaration (MCEETYA, 2008)

Slide 5 - back in 2008 I moved from being a Physics teacher in one school  to eLearning Adviser to many schools, right at the outset of the DER. As such I had Principals, Consultants and Directors asking me what would happen to their exam results now every student would have a laptop? My boss said that sounded like a PhD thesis. At the same time I was offered postgraduate study with the Sydney University Physics Education Research  (SUPER) group following my work with them developing an interactive Thomson's Experiment for HSC Physics as part of the Australian Multimedia for Physics Students (AMPS) program. Accordingly I applied to research the impact of the DER in CEO Sydney through SUPER.

Slide 6 - originally, the main phrase banded around was the National Secondary School Computer Fund (NSSCF) rather than the DER, looking to move to a 1:1 computer to student ratio. There were to be two rounds, 1 & 2, beginning in 2008 and 2009 respectively, operating for 4 years each. However, due to negotiations between DETNSW and the federal government, Round 2 was pushed back to later in 2009 and became Round 2.1.

Slide 7 - CEO Sydney got involved straight away, accessing Round 1 funding for half of its secondary schools and Round 2(.1) funding for the other half. Every year 9 student was provided with a Federally funded Macbook or HP laptop. The schools had to bite the bullet and find funds (not budgeted for) to provided the teachers with the same laptops. CEO Sydney had to foot the bill for wireless infrastructure and tech support (though the DETNSW negotiations may have retrospectively covered this). These devices were issued to every Year 9 student over 4 years to be used until the end of Year 12. As such, some will still be in use until 2015, though no new DER devices have been issued since 2012.

Two unique epochs occurred in 2009 and 2011 where, due to only half of the students receiving laptops in 2008, we had a situation where half of the School Certificate candidature in 2009 and again for the HSC candidature in 2011 had been schooled with 1:1 laptops and half without. These epochs represented unique, never to be repeated, dichotomous scenarios to compare students with laptops to those without. Unfortunately, I was unable to begin my research in 2008/2009 (due to not being Australian enough as a then Temporary Resident!). However, after Permanency (and subsequently  Citizenship) I started in July 2010, and thus was able to capture the 2010 data and importantly the 2011 dichotomous HSC epoch data.

Slide 8 - Visible Learning (Hattie, 2009). Most studies have historically compared classes with and without technology and not different ways of learning with technology. We are investigating both.

Slide 9 - Choosing the Wrong Drivers for Whole System Reform (Fullan, 2011). "Pedagogically vapid" is a great phrase. The full statement is important as it emphasises that student performance is about teaching and learning, not the tech. (Fullan was relatively technophobic up to 2011. However, from 2012, in 'Stratosphere', he sees the opportunities technology potentially offers to teaching and learning).

Slide 10 - Laptops for Teachers: Practices and Possibilities (Cowie et al, 2011) - a more positive quote alluding to the new opportunities potentially provided to teaching and learning with laptops.

Slide 11 - The 3 papers so far from DECNSW on DERNSW by Sarah Howard et al from the University of Wollongong. (I envy the time and hence efficiency with which Sarah can work - my 2010 data analyses are appearing in 2013...). Peter Goodyear who leads the USyd Sciences and Technologies of Learning (STL) network and invited me to speak at CoCo was instrumental in setting up the DERNSW evaluation program and is a mentor to Sarah Howard. 

Slide 12 - The 16 secondary schools I work with and surveyed in South-West Sydney and the Shire.

Slide 13 - School Profiles - quite a spread in SES and ESL, all non-selective.

Slide 14 - Response Rates - very nice :) It helps having a relationship with the schools to encourage participation. Anonymisation of data and an obsession with objectivity will ensure no bias (the results published so far demonstrate this, I want to tell an honest story not butter up my colleagues with faux research). 

Slide 15 - First paper published thus far, second one pending. Results of both discussed below.

Slide 16 - Some questions paraphrasing those in the first paper.

Slide 17 - Questionnaire Tool (using Google Doc Forms)

Slide 18 - Bubble Diagrams for 3 teachers in the same department - 2 very similar and well aligned to their students, one very different and quite misaligned.

Slide 19 - Misalignment Index discussed with results

Slide 20 - Graph of Misalignment Index versus average Likert response for all teachers (split as Round 1 (2 years' experience) and Round 2 (1 year's experience)). The spread and trend are very similar for both groups. Those with higher average Likert responses had lower MI i.e. teachers utilsing and expecting greater use of laptops were more highly aligned with their students i.e. the students follow suit.

Slide 21 - Graphs for Student v Teacher responses for the 4 questions. Greater alignment between teacher expectation of laptop use in class/homework and student reported use (bottom two graphs) than between teacher practices of bringing their laptop to school/class and student practices (top two graphs).

Slide 22 - Empirical Alignment Graph - along with the Misalignment Index, the Empirical Alignment Graph could be used by any researcher when comparing teacher responses/activities with student responses/activities. Do you agree?

Slide 23 - Second paper 'in press' - some of the questions discussed.

Slide 24 - Bloom's Digital Taxonomy (reproduced with permission from Andrew Churches - thanks Andrew!)

Slide 25 - Questionnaire Tool (again using Google Doc Forms)

Slide 26 - Frequencies of use for all applications for teachers and students - note the disparity for Simulations in particular.

Slide 27 - Frequencies of use for 3 most enjoyed - note how students out report teachers for blogging, video editing and podcasting (higher order activities)

Slide 28 - Frequencies of use for 3 used most often - word processing, internet research (googling) and electronic text book reported in that order by both teachers and students (usually low order activities)

Slide 29 - Summary of Findings - note, though few students experience simulations, those that do really enjoy them.

Slide 30 - Summary of Findings continued - note issues with BoS. However, in all of these results it is very important to note that this data is from 2010, only one or two years into using 1:1 laptops for these teachers. Three years on from that the classrooms should look quite different (some researchers will have to investigate :)

Slide 31 - Quote from Halverson and Smith (2009) - a great quote highlighting that in some classrooms/schools/systems technology can be manipulated to simply maintain the status quo.

Slide 32 - Current Research - overview of current analysis of 2011 HSC dichotomous data, directly relating to the student performance (dependent variable).

Slide 33 - Whetting people's appetites with the early results - we have significance and correlation, but not saying what :)

Slide 34 - References for this presentation, many more in the papers.

Slide 35 - Acknowledgements - have Ethics Approval form CEO Syd and USyd and greatly appreciate  the participation and openness from the teachers and students!

Slide 36 - love the quote attributed (though not definitely) to Tagore and the augmented reality pic from turkletom.

With the second paper to be published soon there will be far more detail for readers to cogitate if they so wish. Any dialogue/feedback would be greatly appreciated. 

For me, the highlight of presenting at CoCo was the discussion that ensued during the seminar (and has continued subsequently) with the excellent researchers, educators and student teachers that I met. Many thanks to Peter Goodyear for the invitation, Pati Paez and Maryam Khosronejad for the organising and Martin Parisio for the webcasting.

Deep Analytical Thinking

Recently I was asked to run a workshop on 'Deep Analytical Thinking'. This was essentially a session on Critical Thinking and Higher Order Thinking Skills. The primary remit was to get teachers thinking about what it is they are asking of their students, particularly in assessments. Are they giving students the opportunities and challenges of accessing higher order thinking skills?

Working within the New South Wales Board of Studies (but relevant everywhere) it is particularly important for teachers to know and understand the 'verbs' asked of students in the syllabus, exams and consequently, the assessments the teachers write themselves.

This workshop engaged the teachers in a deep analytical thinking task (whilst getting them to collaborate in a Google Doc); tested their knowledge of the verbs in a fun quiz format (demonstrating the power of Google Doc Forms and Flubaroo script); demonstrated the merit in using unique stimulus data e.g. from http://www.gapminder.org/, thus ultimately allowing for open book/laptop, non-googlable  exams; and discussed Bloom's Taxonomy and Bloom's Digital Taxonomy.

Here is the Prezi, enjoy:

eLearning in Maths

Last week I was privileged enough to work with some amazing Maths teachers (a much-maligned subject area) around eLearning in Maths. The day was essentially a massive eLearning in Maths TeachMeet. We crowdsourced the ideas into the Google Presentation below. This is a working document, very much in the same vein as Tom Barrett's Interesting Ways. Please do peruse and please do add your own ideas.

Why Bother with Google Apps for Education?

Our whole system has gone Google Apps. Below are a couple of slides that people may find useful if they have to sell the idea of Google Apps to colleagues. Best of luck!

Initial data analysis of teachers’ and students’ reported and type of use of laptops in science in 14 Sydney secondary schools

Recently I presented at the Australian Conference on Science and Mathematics Education around some of the early findings in my research, essentially an overview of my first paper plus a taster of the second.

The  presentation, entitled 'Initial data analysis of teachers' and students' reported and type of use of laptops in science in 14 Sydney secondary school', can be found in the conference proceedings here.

The actual presentation can be found in the Prezi below.

1st Journal Paper Published: Seeing Eye-to-Eye on ICT

After the roller coaster ride of peer reviews, editing and rejections elsewhere (see Peer Review, Peer Exposure), my first journal paper has finally been published: Seeing eye-to-eye on ICT: Science student and teacher perceptions of laptop use across 14 Australian schools.

It can be accessed at http://ascilite.org.au/ajet/submission/index.php/AJET/article/view/72/36.

Many thanks to the Editors and Reviewers at AJET.

Please cite as: Crook, S. J., Sharma, M. D., Wilson, R., & Muller, D. A. (2013). Seeing eye-to-eye on ICT: Science student and teacher perceptions of laptop use across 14 Australian schools. Australasian Journal of Educational Technology. 29(1), 82-95.

Peer Review, Peer Exposure

At the end of Pink Floyd's 'The Wall', during 'The Trial', the Judge decrees to the main character Pink that:

I sentence you to be exposed before your peers!

This has some personal resonance as I am currently attempting a part-time 'thesis by publication' thus constantly exposing myself to the scholarly (and sometimes ruthless) trial by peer review.

To recap, I began ostensibly a MSc in July 2010, assessing the impact of 1:1 laptops and the Digital Education Revolution on student performance in science. The MSc was a mutual get-out clause with the agreed intention to ultimately complete a PhD. As such I had to conduct my time as if it was a PhD. This study had been in the planning for a couple of years (while I awaited permanent residency). Consequently, I was fortunate enough to achieve human ethics approval almost immediately and begin surveying students and teachers. With a very healthy return rate this was a great start. The following year was spent reading, learning multiple regression, playing with the 2010 data and collecting 2011 data. I started work on my first paper, presenting some preliminary findings and developing some analytical tools for future use. This was when I first entered the world of peer review.

Towards the end of 2011 was the deadline for submission to present at a very large and famous 'international' (American) edtech conference. In my naïveté I hoped to kill two birds with one stone and submit a research article, thereby contributing to my publishing commitment, and thus warranting my attendance at this gargantuan event (that I had presented at previously ). My superviser wasn't that impressed with the idea but indulged me (what I should have realised was that appearing in conference proceedings has nowhere near the kudos of appearing in a journal). So accordingly I submitted a proposal. This required cutting the 5,000 words I already had down to 2,000 needed to meet the proposal guidelines. Here came my first real mistake - in reducing the word count I decided to whet the reviewers appetite by alluding to my useful analytical tools rather than presenting them for their discernment. Still, it was a decent proposal and I waited with bated breath for the reviewers' comments.

The reviewers were prompt, and the result ... my first rejection. Now, I have no problem with rejection (essential in this game!), but here is the curious feedback. There were 9 criteria, each to be awarded a mark out of 9. The results:

REVIEWER 1 scores: 9,8,7,7,8,9,7,6,8

REVIEWER 2 scores: 9,1,1,1,1,3,3,1,1

REVIEWER 3 scores: 9,8,8,7,7,8,7,6,7

So, overall, Reviewers 1 and 3 seemed to value my paper, awarding 3 perfect 9's and 6 near-perfect 8's between them. Yet Reviewer 2 slammed the proposal with 6 diabolical 1's!!! As already stated, I have no problem with rejection but a 1 is the lowest score possible. This 'peer' considered that in those six criteria I couldn't get any worse, in total contradiction with the other two reviewers. (I have a theory about the identity of Reviewer 2 but it would appear stageringly paranoid!). When teachers engage in pilot marking such inconsistency is not permitted. How is it then that this is allowed to occur for a massive conference? Whilst accepting up front the rejection I put this to the Chair of Review. It turns out that overturning a rejection would require at least two 8's in the final column (Final Recommendation). I had 8, 7, 1. Essentially, I missed out by one mark with the added comments mentioned earlier that I should have provided the analytical tools rather than alluding to them. So endeth Rejection #1.

In the meantime, I had been writing my second paper. This was on a slight tangent, with admittedly low level analysis (comparative frequencies), but interesting findings nevertheless. In early 2012, we found out late in the day about a conference on our very doorstep that might be appropriate for my research findings. As if history was repeating itself (although this time at the instigation of my superviser) I had to chop down my paper into a brief proposal and submit at the eleventh hour. The reviewers were very prompt and well and truly slated the proposal. In the rush, neither myself nor my superviser had quite done enough research into the conference and its conflicting ideology. To quote:

Definite reject: I would argue strongly for rejecting this proposal.

Rationale for the sake of my pride: not much time put into the submission due to time constraints, not too much time lost. Scholarly learning: academics in different departments often have quite opposing ideologies.

After this brutal initiation into peer review, understanding the err in my ways in opting for conferences and actually benefiting from some of the more constructive criticism, I set about improving both of my initial papers for submission to quality journals. After a heavy re-edit of the original first paper, it was submitted to a well considered, regional international journal in February 2012. After a bit of a hiatus the editor replied with recommendations from the reviewers - recommendations not rejections! After responding to every recommendation and query we received some final recommendations which again we addressed. Then, finally, in late August I received this message:

Our decision is to accept your manuscript for publication. Congratulations!

This was sensational news! After the initial setbacks with conference proceedings, I finally had some runs on the board (and good runs at that) with a quality journal. I could now say that I was 'in press'. We were hoping that the paper would make it in time for a 2012 edition but this was not to be. Hopefully the first edition in 2013 will include the paper. This will of course be featured in this blog when it happens.

Whilst the first paper was in review, the second paper was seriously re-edited and submitted to the same journal in May. After the approval of the first paper it wasn't long before the second paper was reviewed. The result - rejection (#3). One of the choice comments was:

this article adds little to what is already known.

To be fair, the feedback was much appreciated even though the right to reply was removed with the rejection, I would have loved to address some of the issues and misunderstandings with the reviewers. Same journal, different paper, different editor and reviewers - different result.

After our previous success this caught us by surprise, but we had the backup plan of submitting to an alternative, recently promoted international journal. I had a good feeling about this submission with the editor being extremely efficient and an open line of communication. However, the final result - rejection (#4). After the initial hope this was quite gutting being rejected for the second (arguably third) time for the same paper. What was particularly disappointing was the minimal feedback from the reviewers, yet still with the choice line:

The title and repeated comments about Australia being the first country to explore 1:1 laptops seem both xenophobic and parochial. 

Australia being the first country to explore 1:1 laptops over 20 years ago is simply a well published fact. Any article about the Australian Digital Education Revolution and 1:1 laptops has to mention Australia, a lot. At worst this could be considered 'parochial' (a common feature of Australia and Australians) but in this instance I would argue it is simply the relevant context. I had thought a journal that includes articles on Taiwan and Singapore would also be interested in an Australian story. As for 'xenophobic',  I took issue with this and received a formal apology from the editor and publishers. Whomever this peer and reviewer is, he/she does not know the meaning of the word. As the editor highlighted in his apology, if there had been even a whiff if bigotry or racism it would have been rejected out of hand immediately.

So the current situation is that my second paper has been resubmitted yet again, this time more locally with a good chance of success (my superviser is the editor!). I am currently working on the 'biggy', a multiple regression analysis of the unique 2011 data (all of this resubmitting and re-editing has put me behind in the prompt reporting of findings from the data). I did finally upgrade to PhD, but only after my paperwork was lost and the resulting resubmission being briefly rejected by 'new protocols' that surfaced in the interim.

In conclusion, I do totally understand and value the process of peer reviewing. In fact, if society, particularly the tabloids and shock-jocks, understood the merit and subsequent quality thanks to peer reviewing we would not have such inane, ignorant arguments and diatribe regarding much studied issues e.g. global warming. That being said, the necessary discrimination in peer review leaves many of us licking our wounds and sometimes perplexed by the reviews. As my good friend @effectsofNAPLAN pointed out, both teachers and researchers need a thick skin ... and any teacher involved in research requires "crocodile thick" skin :-)

It is important to note that reviewers do so voluntarily in their own time. This is an honourable task that I hope to aspire to one day, when I truly consider myself a peer. In the meantime, I endeavour to improve the quality of my work and take any knock-backs on the 'chin'.