Citizen Science session, CSCW 2013

ACM Conference on Computer Supported Cooperative Work and Social Computing
27 February, 2013
San Antonio, TX

Citizen Science session

——

Sunyoung Kim – Sensr

Intro to types of citizen science, diversity of project types. Common underlying characteristic: using volunteer’s time to advance science. Many typologies, projects can be divided by activity types into primarily data collection and data analysis/processing. Focus here is field observation, has great opportunities for mobile technologies.

Problem is that most citizen science projects are resource-poor and can’t handle mobile technologies on their own. Goal is supporting people with no technical expertise to create mobile data collection apps for their own citizen science projects. Terms used: campaigns – projects, author – person who creates campaign, volunteer – someone who contributes to collecting data/analysis.

Design considerations include: 1) current tech use, similar available tools, needs for practitioners. Reviewed 340+ existing projects (campaigns) from scistarter.com, found only 11% provide mobile tools for data collection. Looked at types of data they’re collecting – primarily include location, pictures, and text data entry. 2) Data quality is paramount, and data also contains personal information. 3) How to recruit volunteers. Looked at similar mobile data collection tools like EpiCollect and ODK. They’re pretty similar in terms of available functionality, but Sensr is simplest to use. Most comparable platforms are open source so you need programming skills to make them work (free as in puppies!) – even the term open source can be very techie to the target users.

Built Sensr as visual environment combined with mobile app to author mobile data collection tools for citizen science. Demo video demonstrates setting up data collection form for “eBird”, pick fields to have on form. Just a few steps, creates back end database and front end mobile interface. Very straightforward interface to assemble a mobile app for citizen science data collection.

A couple of features: can define geographic boundary but can’t prevent people from outside the boundary to join (App Store is global), but you can help users target correct places. Can review the data before it is publicly viewable or goes into scientific data set.

Did case studies to see how nontechnical users did with it, betas with existing projects, before launching tool. Strong enthusiasm for the app, especially for projects with interest in attracting younger participants. Main contribution: Sensr lowers barriers for implementing mobile data collection for citizen science.

Question about native apps versus HTML5 mobile browser apps due to need for cross-OS support.

Question if there’s a way to help support motivation; not the focus in this study. Case study projects didn’t ask for it because they were so thrilled to have an app at all.

——

Christine Robson – Comparing use of social networking and social media channels for citizen science

One of main questions from practitioners at Minnowbrook workshop on Design for Citizen Science (organized by Kevin Crowston and me) was how to get people to adopt technologies for citizen science, and how to engage them. They were questions that could be tested out, so she did some experiments.

Built simple platform (sponsored by IBM Research) to address big picture questions about water quality for a local project, and this app development was advised by California EPA. App went global, have gotten data from around the world for 3 years now. Data can be browsed at creekwatch.org, you can also download it in CSV if you want to work on it. “Available on the App Store” button on the website was important for tracking adoption.

Creek Watch iPhone app asks for only 3 data points: water level, flow rate, presence of trash. Taken from CA Water Rapid Assessment survey, used those definitions to help guide people on what to put in the app, timestamped images, can look for nearby points as well. More in the CHI 2011 paper. Very specific use pattern: almost everyone submits data in the morning, probably while walking the dog, taking a run, something like that.

Ran 3 experimental campaigns to investigate mobile app adoption for citizen science.

Experiment #1: Big international press release – listed by IBM as one of the top 5 things that were going to change the world. It’s a big worldwide thing when IBM makes press releases – 23 original news articles were generated, that’s not including republication in smaller venues. Lots of press, could track how many more new users came from it by evaluating normal rate of signups versus post-article signup. +233 users

Experiment #2: Local recruitment with campaign “snapshot day”, driven by two groups in CA and Korea. Groups used local channels, mailing lists, and flyers. +40 users

Experiment #3: Social networking campaign: launched new version of app with new feature, spent a day sending messages via FB and Twitter, guest speaker blog posts, YouTube video, really embedded social media campaign. Very successful, +254 new users.

Signups aren’t full story – Snapshot Day generated the most data in one day. So if you want more people, go for the social media campaign, but if you want more data, just ask for more data.

Implemented sharing on Twitter and Facebook – simple updates as usually seen in both systems. Tracking sharing feature – conversions tracked with App store button. Can’t link clickthrough to actual download, just know that they went to iTunes to look at it, but it’s a good conversion indicator. Lots more visits resulted from FB than Twitter, a lot more visitors in general from FB as a result. Conversion by social media platform was dramatically different – 2.5x more from FB versus Twitter or web, which were pretty much the same.

Effects of these sharing posts over time – posts are transient, almost all of the clicks occur in the first 2-5 hours, after that its effect is nearly negligible. Most people clicked through from posts in the morning, there are also peaks later in the evening when people check FB after work; then next morning they do data submission.

However, social media sharing was not that popular – only 1 in 5 wanted to use Twitter/FB feature. Did survey to find out why. Problem wasn’t that they didn’t know about the sharing feature, 50% just didn’t want to use it for a variety of reasons. Conversely, for those uninterested in contributing data, they were happy to “like” Creek Watch and be affiliated on Facebook, but also didn’t want to clutter FB wall with it.

Facebook campaign as effective – or more – than massive international news campaign from a major corporation (though the corporate affiliation may have some effect there), and much easier to conduct. Obviously there are some generalizability questions, but if you want more data, then a participation campaign would be the way to go. Sharing feature shows some promise, but it was also a lot of work for a smaller payoff. With limited resources, it would be more useful to cultivate Facebook community than build social media sharing into a citizen science app.

Trip Report: Workshop on Human Computation for Science & Computational Sustainability

Human Computation for Science and Computational Sustainability workshop at Neural Information Processing Systems conference, 12/7/2012, Lake Tahoe, NV.

———

Workshop details: http://www.cs.cornell.edu/~damoulas/Site/HCSCS.html

Timo Honkela also posted great summary notes, which are much briefer than what follows…

———

Keynote speaker: Eric Hortvitz, MSR

HCOMP and AI: programmatic access to people. This generates new opportunities with applications in science, sustainability, and society.

Programmatic access brings people into machine intelligence. We can apply this to construct case libraries for learning classifiers; debugging and refinement of perception and reasoning; human expertise as components of larger systems; probe frontiers of MI competency; and addressing “open world” challenges.

Classic example is ReCAPTCHA. Using AI to optimize HCOMP – mutual construction. Coherent fusion of contributions – predictions, measures, recommendations; guiding human effort via active learning, expected info value; ideal plans for decomp & sequence of efforts; incentivizing design.

Study in citizen science to combine ML and decision making at MSR. Came out of a conversation w/ Jim Grey about 15 years ago. Focusing on consensus tasks in citizen science – classic crowdsourcing. Questions: how to fuse automated analysis with humans, whose vote counts more, etc. Ran into Lintott when Galaxy Zoo was starting. Describes how GZ works and how awesome it is, what they’ve discovered, etc. Core science is coming out of citizen science.

CrowdSynth: machine learning for fusion and task routing; learning from machine vision and votes. Combines machine & human perceptions – different sources of cognition and intelligence. GZ is using people, sole focus, but SDSS has applied machine vision to all of these data for about 450 features, very specific details that are completely incomprehensible to non-astronomers, e.g. “isophotal axes.” With both of these information sources, can we combine them for machine learning, predication & action?

So layering human abilities onto machine abilities, depending on cost and ability, using them together to get the most out of the complementarity of the systems. CrowdSynth gives a task that machine figures out how to assign, humans do their thing, etc. What’s inside the box: machine features and cases w/ task and worker databases. That feeds into answer models and vote models, both of which go into the planner. The task features are for machine vision. Worker features involve experience and competency – employees can be tracked – dwell time, experience, accuracy. Vote features – distributional aspects of votes on tasks – # votes, entropy, mode class, vote of most accurate or experienced worker, etc.

Vote models predict the next vote, and answer models predict correct answers. They used a Bayesian model selection to construct models that can predict outcomes – looks like 100+ variables. Model sensitivity to number of votes – answers require 5-9 people; votes require more like 50 people.

Soul of the system: planning goal to optimize hiring decisions to maximize expected utility. Consensus tasks are modeled as finite-horizon MDP with partial observability. Long evidential sequence problems – each incremental vote/answer is worthless until they’re aggregated at a certain point [pooled interdependence, could apply coordination theory?] The value of additional human computation was simulated – fairly complex process. It worked really well – new efficiencies and better stopping criteria to maximize accuracy. Decision theoretic methods ideally route tasks to individuals, can get better results with fewer people.

Did real-time deployment within Zooniverse project, Planet Hunters. First test of realtime system that’s not based on retrospective data. Moving beyond consensus tasks, the Milky Way project – people labeling images for “interestingness” – should be really interesting to see how that works. Predicting and promoting engagement – how to engage volunteers and figure out when they will disengage. Goal is designing interventions for engagement. Trying to figure out how many tasks before they disengage, want to learn about short versus long sessions and how long they disengage between sessions. Given training data, can we predict when they will disengage?

Features in engagement models. Task features include mean agreement and related metrics; session features include log-based metrics, e.g. time on task (very interesting inferences to be made); long-term historical focuses on things like total experience of tasks, # sessions, accumulated info about a person based on their participation.

Found that session and history features more important than task features or session features alone; found average engagement of 4 minutes and 20 tasks – bite-sized participation. Also working on rich models of individuals abilities – things like user activity and experience to figure out accuracy. Personalizing delivery of tasks by current skill level will also let them tailor skill development if there’s a specific pattern of errors.

Examples he finds exciting to leverage in-stream activity of crowds. Ambient data (trace data) from crowd behavior in science and sustainability – like eBird SoCS project. Example of Lac Kivu earthquake in 2008. Looked at cell communications data on 6 days around the event, used it to detect anomalies as earthquake hit. The predicted epicenter was a few km from true epicenter. Inferring opportunities for assistance based on % increase in calls. Had coherent measures of uncertainty for optimizing emergency response. Could do really interesting things with intentionally collected data, not just trace data.

Example: Identify drug interactions with the crowd. FDA adverse event reporting system (AERS) because pharmaceutical companies do limited testing. But in the wild, they’re finding new interactions: Paxil and Pravachol don’t cause hyperglycemia individually, but they do in combination [metabolic syndrome]. Can use large-scale web logs on searches related to “pharmacovigilance” – people searching on hyperglycemic systems along with drug names in queries. Do disproportionality analysis of reporting rations – observations versus expected in Venn diagram style, high statistical systems.

Final example: Crowd physics [crowd coordination] – coordination of people in space and time. Collaborations and synchronization in time and space – flash search & rescue team, transport packages between arbitrary locations quickly with low-cost hand-offs. Disrupt flow of disease from epidemiological models. Used geocoded Tweets to identify feasible coordination by distance and time, can think about how to incentivize to change those limits. Changing slack and wait time for coordination; use small-world opportunistic routing; incentives to modify graph properties.

Animated visualization of individuals’ movements based on tweets and their location change between tweets. Found a lot of tweets at airport hubs, realized they could route most packages in 3 hours that way.

Summary: learning and inference for harnessing human & machine intelligence in citizen science. Great opportunities for doing science with crowd & potential to coordinate crowd on physical tasks as well as virtual tasks.

———

A meta-theory of boundary detection benchmarks, X. Hou et al., Caltech & UCLA

Looking at how to find edges in images, drawing the lines on a photo. Huge variability in detail they included, which isn’t being addressed by current computer vision benchmarks. Increasing label consistency has to do with looking at which labels are identified by everyone, and which are only detected by a few (orphan labels.)

Did experiment to test the strength of line segments, whether human boundary is considered stronger by a third party, specifically the orphan labels – “false alarms” – like grasses highlighted in a photo of a pheasant. Algorithm will identify false boundaries based on that, developed a way to evaluate risk of false positives by subject number – orphan labels are about as strong as machine labels, which is a big problem.

Type I/Type II errors – believe all existing labels are correct, can always rationalize the boundaries behind it, but there are a lot of misses – labeling weak boundaries but not strong ones, while algorithms highlight everything. How to identify the ideal overlapping regions between orphan labels and machine labels? Gets a bit technical about how they approached the question, but managed to use an inference process to substantially improve boundary strength over initial noisy data. Another experiment worked in reducing risk.

———

Evaluating crowdsourcing participants in the absence of ground truth, R. Subramanian et al., Northeastern University, LinkedIn & Siemens

Problem setting: supervised/semi-supervised leaning – ground truth exists but not available or expensive; multiple sources of annotation. Question is evaluating annotators – are they adversarial, spammers, helpful? Example uses: identify helpful/unhelpful annotators as early as possible; evaluate data collection/annotation process/mechanisms.

Using binary classification; not all annotators label all data points, and ground-truth not available. Example scenario: diagnosing coronary artery disease, much difference in cardiologists’ expert diagnosis. CAD can be diagnosed by measure and scoring regional heart-wall motion in echocardiography; quality of diagnosis highly dependent on skill & training; increasingly common scenario.

Challenges: variability in annotators – comparative reliability between people, internal reliability by person, maliciousness. Practical questions for healthcare – how to diagnose if docs don’t agree, how to tell which docs are skilled enough.

Multi-annotator model – gets into much more technical detail with probabilities for graphical model. Read paper for more details.

———

Using community structure detection to rank annotators when ground truth is subjective, H. Dutta et al., Columbia University

Chronicling America project – National Endowment for the Humanities and Library of Congress project. Developed online searchable database of historically significant newspapers from NY Public Library collection between 1830 – 1922. Question of how to improve indexing and retrieval of the content. Many notable events. Describes how historical newspaper is made searchable – scanned, metadata assigned, OCR.

Data pre-processing involved NLP and text mining, then similarity graphs for articles with edges based on cosine similarity of TF/IDF beyond a certain threshold. Choice of threshold will generate multiple ground truths, however, so subjectivity is introduced even in this automatic process.

Did community structure detection using modularity maximization; that’s an NP-hard max-cut problem. Approximation techniques are therefore the best approach.

———

Crowdsourcing citizen science data quality with a human-computer learning network, A. Wiggins et al., DataONE, Cornell Lab of Ornithology

Slides

———

Human computation for combinatorial materials discovery, R. Le Bras et al., Cornell University

Goal is developing new fuel cells – current electrocatalyst is platinum but still not great and way too expensive. Process for finding alternatives is rather technical. Using CHESS to identify the resulting crystal structures, resulting in graphs of the underlying lattice on the silicon wafers. The experiments can be run for about 2 weeks a year, costs $1M/day to use CHESS.

Satisfiability Modulo Theory approach. It all gets very technical.

UDiscoverIt UI: download a client, several complex data displays for pattern identification. Select slices, look for patterns in the heatmap of “Q-values”. Include user input speeds up the process by 1-2 orders of magnitude, despite only involvement in a minimal part of the process by providing useful information about the structure of the problem, which reduces the search space for the SMT solver and improves overall performance. Lots more to do if they’re going to get this to implementation, including getting it to a point where they can run it on AMT.

———

Dynamic Bayesian combination of multiple imperfect classifiers & An information theoretic approach to managing multiple decision makers, E. Simpson et al., Oxford University, University of Southampton & Zooniverse

Human computation: people are unreliable; they either learn or get bored. How to optimize the crowd, maximize scale, and maintain accuracy? Zooniverse is their example domain – people do pattern recognition tasks and label the objects. The problems they’re addressing involve tracking worker reliability, combining classifications, and mixing their computational agents where possible to assist workers and scale up.

Probabilistic model of changing worker behavior – treats artificial and human agents and base classifiers, conditionally independent responses given the type of object. Uses Bayesian approach to combining the decisions – dynamic extension of IBCC incorporating prior info like known expertise. Faster version with Variational Bayes, semi-supervised and dealing with limited training data.

Technical details with Dirichlet distributions and the like.

———

Building the Visipedia Field Guide to NA Birds, Serge Belongie, UCSD

Working with CLO; this is a status update on a bird recognition system. Visipedia is a visual counterpart to Wikipedia. Subordinate categories for recognization. Similar in some ways to Leafsnap. Lots of crowdsourcing involved at multiple steps; breaking down something really complicated into smaller easy tasks. Oh, wait – it’s Merlin!

Processing images – MTurk – labeling attributes. MTurkers liked it so much that they complained when they took it down! There are no accounts or scores or anything – the reward for finishing the task is getting another task! People like pretty flying things.

“Taster” sets – bitter vs sweet – pretty colorful birds like scarlet tanagers, vs “little brown jobs”. Need to give people pretty things sometimes. Setting up as a visual 20 questions. Current iPad app – 200 species, pick a bird. Films strip on the left, sorted the 200 species in order of likelihood of match to a photo taken by the user. Trying to find the bird parts that people click, heatmapping where the computer thinks the body parts are.

Trip Report: USGS CDI Citizen Science workshop, day 2

9/12/12 USGS Community Data Integration Citizen Science workshop

Ice Core Lab, Federal Center, Denver, CO

Data Management session

——

My talk: Data Management for Citizen Science: Challenges and Opportunities for USGS Leadership

——

Austin Mast, Florida State University, iDigBio

Public Participation in the Digitization of Biodiversity Collections

iDigBio is national resource for advancing digitization of biodiversity collections. Key objectives include digitizing data from all US biological collections, large and small, and integrate these in a web accessible interface using shared standards and formats – huge challenges b/c estimates suggest about 1 billion specimens in bio collection at thousands of institutions, and 90% are not accessible online. Community collaboration and technology are clearly important to this. Only one mention of public/citizen in the document – may contribute to digitization workforce.

Public participation is going to be necessary to accomplish this in 10 years. Produced strategic plan in 2010 from 2 workshops and that yielded an NSF program (ADBC) which is collaboration between Bio and Geo directorates to fund digitization of thematic collections networks (TCNs) on RQs, and national hub to coordinate. Currently have 7 TCNs, example of New England Vascular Plant Specimen Data to Track Environmental Changes project. Goal is 3 centuries of data with about 1.3 specimens and images from herbaria in New England. Focus of other TCNs include lichens, plants/herbivores/parasitoids, arthropods, macrofungi, vascular plants, fossils, and integrative platform development. Geo distribution of 130 institutions currently involved in these groups – all across the country.

Goals of iDigBio is enabling digitization, portal access in cloud environment, engage users in research and outreach, and plan for long-term sustainability. Initially 5 years, $10M, if all goes well, possibility of another 5 years. Project is not going away soon, means they can collaborate more intensely. There is a PPSR-like WG in iDigBio that is working on engaging the public in digitization at earlier stages than just data use. It cuts across their components of digitization, CI, research, education & outreach.

Models for digitization processes – getting data to distribution points, classifying the different workflows into three models. Have since focused on types of specimens being digitized and how that intersects with workflows. Hope that public can be engaged in specimen curation & imaging, text transcription and specimen description from images, & georeferencing. They do specimen curation w/ about 3 volunteers per semester, requires on-site presence. The other two tasks can be done in distributed format.

Examples of text transcription projects include Apiary project, volunteers ID regions of interest in the image (draw rectangles around specimen label and calling it a label), then those images get fed to volunteers to transcribe it to correct OCR or straight-up transcription and categorizing the elements e.g. who collected it and where. Specimen descriptions from images are also realistic, i.e. in classrooms. Games are another way to do it, with example of Forgotten Island.

Georeferencing tasks have been done with students for records for their own counties due to familiarity. They use GEOLocate to annotate map layers, very successful so far. Often observations are collected repeatedly in the same spot, don’t want to georeference over and over. Volunteers use zooming tools to specify precision of locations based on records.

Challenge: Provide opportunities for not just contributory participation, but also collaborative and co-created participation. Big challenge but worth taking on. Example of CitSci.org: what CI is needed for this community to build a historical dataset of relevant specimens from Milwaukee by digitizing target specimens from across US collections? And what is necessary for sharing the info to CitSci.org as it is constructed? Want to allow comparison of current data to historical data.

What CI is needed for a student to gain recognition for volunteer hours participating in this kind of science? What is necessary for her to gain service learning credit, e.g. for President’s service award?

iDigBio has a special role it can play – the cloud-based strategies are the resting place for the data, thinking about it earlier on in the workflow to make it a source for content that needs to be worked upon.

——

Mike Fienen, USGS Wisconsin Water Science Center

Social.Water and CrowdHydrology

Audubon Christmas Bird Count started as response to shotgun ornithology. Participation has grown tremendously over 100 years, and the data are meaningfully used in science. He runs groundwater monitoring network, with over 100 wells regularly monitored but half by citizen observers and has been going on for a really long time – they used to submit data on postcards, now by email.

Harnessing crowd for scientific data and analysis – his ideas sparked by wildlifecrossing.net which uses photos of roadkill to figure out where animals cross roads. Found CreekWatch and thought they did it already, but not exactly, and they didn’t want to rely on smartphones due to a number of issues and wanted low barrier of entry for both themselves and the participants. Only need to send a text message to participate in CrowdHydrology project.

Guy at Buffalo set up a little infrastructure for 10 sites he needed to monitor, put a note on the sites to text him a message of water levels, that he manually put into a spreadsheet. Mike helped develop software stack for this using Google voice, imap and python using stuff that’s already there with standard email protocols, people can text a local number with Google voice, the system automatically logs into the account, checks the data and parses them into a csv and plots the data in almost realtime on the web. Important for this to be open source to share – not building real infrastructure but building on existing data.

Basic signs included no attempt to tell people how to format messages – afraid if they said to type in station number and value, they would type in exactly that, weren’t sure how it would work for interpretation. They’ve since improved the signs on the gauges. There are also shore signs with more detail that they worked pretty carefully on making sure it was understandable. Worked to make sure there was no language of art – picture of guy “find the ruler” to measure “water height” (not water level, that confused people.) Sign tells people that they can see the data point within minutes, so people can look at data almost right away to see their own data points and start looking at trends.

Cost of generality: “fuzzy wuzzy” – texts came in many formats, some very basic, others very descriptive. Got a few messages that stations weren’t getting NY, but By or My due to iPhone autocorrect and fat fingers with proximal values. Why no typos on y? Found research on favorable/unfavorable accuracy so now they understand typos better.

They use regex to trim out irrelevant info after checking that they contain some of the keywords, using FuzzyWuzzy open source code to make it a lot easier, made by a ticket scalper, found in 2 minutes online, easily implemented. They now use better regex to find the value after identifying station number. Shockingly sophisticated database (csv file) with four fields – date/time has some drawbacks in case data submitted later. Data integrity something of an issue – incorrect observations quite obvious but not removed from data for several reasons. Validated the station with transducer data, verified that the American public can read a ruler. Records for precipitation incorporated, saw major rainfall event lining up with increasing set of measurements.

200 values at site NY1000, but max of 8 observations at other sites. Why? People go there to check out beavers, and it’s near a nature center so they’re primed to participate. Other locations are trout fishing holes that get a lot of visits, had one person at a bait shop ask about whether anyone participates – said wife walks by every day and wondered if anyone contributed data, but has never done so herself!

Collaborating with social scientist to get a grip on social aspects, found Trout Lake LTER to work on this together, they’re already doing surveys and such that complement his skills. Future plans also include looking at lakes and streams in the glacial aquifer system. Other plans – publishing papers, Social.Water code available on github.

Handling the data – using USGS cycle, have really focused on first four steps, haven’t found good ways to validate all data points, have focused on validating process for now. PII considerations play into storing contact info, and because of that the data doesn’t live at USGS but instead at a university. Found another group already planning to do it, so if they had waited for bureaucratic approvals, they would have been behind. Asking about value of recruiting trained observers, funding level from inception to two papers was zero, did it on his own time. Were criticized in paper for not having town meetings and training people, but that’s not free!

Crowdsourcing hydrologic info may be secondary source of data – $100 per site investment rather than $20K instruments – but is a primary source of public engagement.

——

Tim Kern, USGS Fort Collins Science Center

Framework for Public Participation GIS: Options for Federal Agencies

Working with a variety of DOI agencies to help them engage public in routine monitoring efforts. Need is obvious, resources keep decreasing, responsibility keeps increasing, and office turnover means that organizational memory is faltering. Impediments are PRA, PII, technology policies – especially TOS reviews and security and purchasing blocks, and data integrity requirements. The fact that we’ve seen so many examples of work-arounds tells us there are big issues.

Have put together framework for implementation within policy. PRA limits inputs to general comments on an identified area. PII/PIA is handled with hashing personal info. Tech policy means writing for mobile web and using approved social media APIs. Data integrity requirements – they’re working toward developing metadata and starting to connect to USGS publication process. Really do need ombudsperson to enable the work.

Their workflow is being implemented across several agencies. Workflow includes elements such as: study data, catalog and repository, advertise study, block for social media/custom web/mobile option, publication, & reporting. Repository for the data, building the pipes for getting the data from those sources so others don’t need to do it.

Starting off with developing study metadata. Then working with systems built around secure enterprise repo – currently using ScienceBase with agency-specific portals, and point people to other repos like DataBasin. ScienceBase has full DOI security, provides spaces for multiple projects/studies within projects, etc. Study area is loaded into repo, example of complex map data, gives view for public comment and interaction. System redelivers in web services, so there are other service endpoints.

Needed to make sure agencies could work w/in their own agency contexts and systems. They can put their products through an approval workflow w/in USFS, for example, gives a lot of flexibility in developing good materials. Once data is in repo, it can go to client device for comment.

Data collection marketing is critical – it’s not a build-it-and-they-will-come thing. Outreach guidelines include community meetings, local media, social media, agency websites. Also need to advertise access options – hashtag publication, site URL, app download. Borrowed from USGSted for incorporating data from Twitter, ripped off ideas from others w/in agencies to build something usable and useful.

Helped write clients for public data input options – multiple screen sizes, limitations of mobile views, need native app for full capability – especially important for offline data collection. That’s a problem because right now USGS doesn’t allow it, but USDA has approved Apple Store, so they can send the software to USFS because USFS are permitted to use it and distribute it. Automated input processing – Twitter term harvest, location inference, user metadata obfuscation. Web/mobile automation involves comment parsing, logic/term identification, comment screening, image metadata scraping, location and spatial data capture. A lot of high schools have blocked social media, but they can access USGS, and students have found this and are using it to do their Facebook updates.

Data management and distribution: Working on data publication with metadata development and approval workflow. For data discovery and distribution – search, preview, feedback/analytics, download, multiple service endpoints back to the data. So the data is getting out there.

——

Derek Masaki, USGS CSAS, Eco-Science Synthesis, National Geospatial Program

Vision for Data Management in Support of Future Science

Data coordinator with BISON project, looking to increase his participation in citizen science. Send a species observation to @speciesobs or speciesobs@gmail.com.

TechCrunch wants to disrupt government IT priorities. Strategy points: open data will be new default; anyway, anytime, any device; everything should be an API; make government data social; change meaning of social participation. Instead of treating people as though they’re going to muck up the data, start thinking about leveraging those resources more productively.

Embrace change, explore disruptive technology! Need to fight misconceptions – social/volunteered info is inferior, lots of evidence that this is not true, e.g. comparison of Brittanica and Wikipedia, Google bought Frommers for $20M but Yelp market cap $1B – all crowdsourced info, cognitive surplus that we have because we’re a privileged society. Online info can’t be trusted: obviously not so w/ Wikipedia studies. Who would read a book on an iPhone? We want to do everything on our smartphones!

What do do about open data: stop hoarding data! The crowd will make it a superior product and find better uses. E.g. GMaps base data source of TIGER US Census Bureau data, improved through millions of edits, ground truthing, image algorithms – USGS is great at scientific data, but bad at enterprise software development, so stop doing it! Push data out, let others figure out what to do with it and add value.

Changing social participation: empower the crowd! Power to the people – how many scientists in USGS? 10K employees in 400 locations, compared to 55M K-12 students in 100K schools.

Embracing change with disruptive technology – handling massive participation through volunteer county network. Pick 3K US Counties, 25 schools per county, each school posting four records per week – every month it would generate 1.2M data points! People really do want to do this, so let’s involve them.

Data system framework – source data to dataset registry, then to workflow processing for derived data index, curated data, and then data delivery through HTML, REST, SOAP service APIs (could push to SciencePipes!) Does his software development in his free time and isn’t great at it, wants others who are good at it to make it good stuff and make it salable and usable.

Citizen science is about people – need to invest in capacity, technical training, building monitoring networks (Marine Survey, Kauai Algae Restoration, Coastal Dune Restoration in Maui). Thinks kids are best resource – teach kids how to do quality data and monitoring, and they will keep doing it throughout their lives. This will help address constantly decreasing budgets.

Think big, consider data management and human management. Can coordinate volunteer network of 10K, conduct biodiversity survey of every US County, generate 1M obs records next year, develop mobile bio observation form standard, implement standards and generate resources.

———

Technology & Tools session

——

Jen Hammock Encyclopedia of Life (WebEx)

The Encyclopedia of Life as a Source of Materials and a Venue for Showing off Your Work

Intro to EOL – open access materials aggregated from many sources in a public venue, with context and provenance, plus species-level info and also higher taxa. What they don’t do is archival storage, or specimen-level or observation-level info.

Diveboard – program using info from EOL to let divers create dive lists! Intro to Scratchpad – difficult learning curve but powerful. You can enter your own info for any species, load up taxonomic structures from EOL, etc. Working with other API users who need image, video, and text content.

Other projects they’re pushing out include educational tools and apps – Field Guides to customize your own field guide as desired – potentially very helpful for many projects. Any export of content is public domain info and CC info, which does come with license requirements; there are no ND content in their content so you can actually do what you like with the images so long as you give attribution.

EOL has a lot of resources to use the content. Next set of functionality is posting content to pages – CC still applies, Flickr pics in EOL groups. Quality of contributed images are pretty high, Flickr group is most prolific content source. Many sources of images being shared, iNaturalist just got added and others in the works, Morphbank is another one for scientific images – good bulk upload tool, properly licensed content automatically goes to EOL. Videos from YouTube and Vimeo as well, working on SoundCloud for audio records. Not much direct upload of media, but other platforms are better at it so they haven’t rebuilt tools that are already working quite well, just connecting to them instead.

A similar content partner that isn’t biodiversity data is Encyclopedia of Earth (eoearth.org), they are cross-linking taxa to habitats across the two platforms, so river info currently in EOL and linked to all species that occur in that area (Amur River Benthopelagic Habitat).

Review content varies by partner, so although iNaturalist info has already been verified and comes in as trusted – but curators can change the status. Curators’ favorite activity is looking at unreviewed content, so depending on the project they ask for a judgment call as to whether the data come in as either trusted or unreviewed.

——

Jessica Zelt, USGS Patuxent

The North American Bird Phenology Program: Reviving a Historic Program in the Digital Age

Program started w/ Wells Woodbridge Cooke, really into bird migration and made it a research project in 1881, got friends to record arrival/departure dates in their areas. Started w/ 20 participants, but AOU was founded and network let them grow to about 3K observers. Program ran successfully for 90 years, never had a name, Chandler Robbins closed it in 1970 to focus on NA BBS. Many original observers were some of the most notable naturalists and ornithologists of their time, but also ordinary citizens – diverse contributors. Collected 6M records, everything that was known about bird migrations at the time, including publications, breeding and nest records, records of extinct and exotic species. Contributed to AOU checklists and first field guides.

Records stored in 52 filing cabinets in leaky attics and basements, then offsite storage facilities, the records got forgotten. Chan prevented the destruction of the records, many were actually stored in his house. In 2009 funding was acquired to hire a coordinator and scanner to revive the program because it could be used for tracking climate change as baseline data.

Cards were scanned for a full year before they went live with a website and data entry page. Goals were to curate, organize, prioritize data records, scan and key cards w/ QA, create digital format for data, generate and develop network of volunteers to do record transcriptions, automated transcription verification system, etc. Very ambitious.

Workflow is scanning of images to PDF, retained as raw records, images then sent to website, people signup and watch 15 minute video and then they can start transcribing. Card formats differed across program coordinators – originals were hand transcribed to notecards from mailed records. More formalized data formats were created later that didn’t require manual transcription. Cards look different but contain same info.

Constantly refined data entry interface, most important part for motivating contributors is “My Stats” bar that shows transcription numbers – for session, individual, all. Database structure is pretty simple overall. Pretty cool system for transcription verification – if first two transcribers don’t match, third transcriber takes a shot; if still no match, then it goes to “rectification system”. Calling it a “validate-o-rama”.

Volunteer recruitment: tried lots of strategies, got most people from a press release that was picked up by a number of content-related venues plus ABC. Volunteers are from all over the world, though 80% in US. Retention strategies: let people choose own level of involvement, establish a community by interacting, give a reason to donate time, allow them to feel needed, keep lines of communication open proactively, satisfaction survey, allow contributions of suggestions and improvements anytime, and recognize each volunteer for their work. Monthly newsletter to give feedback, announcements, news, a look ahead at what’s coming, plus a volunteer of the month who’s been a strong contributor (they write their own profile). Also picks observer of the month, does write-up about them, and that helps create a connection across time.

Includes a trivia question that isn’t a quick Google, has been really popular and usually gets an answer very quickly. Has also supported competition with leaderboards (typical long tail distribution) and names shown as handles. Very popular, have had to add/update “top” transcriber lists because people are so into it. Woman in 1st place has transcribed 150K cards, but takes time off for bird banding – and sometimes does transcription during banding!

Clear annual patterns of transcription on monthly basis. So far have scanned over 1M records, have transcribed about 700K and have 120 participants. All data released on the website, soon releasing about 160K validated records, making headway with publications as well. System is being repurposed for crowdsourcing in museum collections, citizen science, other large datasets. Other future goals include depositing in repositories, either start collecting migration dates or merge with another program. Also starting to work on “stomach content cards” but it’s a lot more complicated.

——

Derek Masaki, Sam Droege, USGS

Mobile Application use in the 2012 Baltimore Cricket Crawl

Twitter-based citizen science observation platform, initially in NYC, then Baltimore/DC – very successful with mobile platform, soon to be replicated in Hawaii. Web-based viz tool with citizen science info from Twitter client & API, Twitter-based submission protocol, scripts mine Twitter stream API, then output mapping, tabular data.

Ran project as an event, could submit data with Tweets, email, SMS, voice, got lots of media coverage. Only 8 species to look for, one minute survey, wherever you want. Send results right from the site, HQ processed the info as it came in and then mapped it in real time on Discover Life and U of Hawaii. Got 400 sites surveyed by 300 individuals, still getting submissions for phenology data to see when cricket calls fall off. About 1800 species observations, most went out in small groups, 75% of data came in email, mostly from mobile, and 10% from Twitter so that was worthwhile.

Data sheet simple – shortcodes for the 8 species w/ space for counts, name/date/start time/location. Tweets pop up on the map in the correct location with optional image/sound files attached. Used to have problems with audio filtering that removed high audio range and made it impossible to use smartphones for recording crickets, but now it’s no problem and they use a free app to collect and upload the audio clips. Future goals – phenology program regionally; national coordination of volunteer monitoring of singing insects; reuse technology for frogs and birds; Hawaii K-12 Critter Crawl in October, and USGS support for social media/mobile. Reuse of tech works well because frogs stop calling right about when crickets start up. Currently operating through Google Maps and Gmail, hosted by U Hawaii because they can’t run it off USGS as yet. Eventually want to do some audio vouchers on it, and finding that observers are accurate through ground truthing and they don’t report questionable IDs.

Findings – these species haven’t gotten attention in a century b/c scientists didn’t think it was that interesting. Wanted to learn if certain katydid had been extirpated, turned out that there were 7 species they thought were extirpated but were actually still around. Jessica did it, found it easy, and liked that it used systems that people were already familiar with – plus “cool factor.” Can also verify with waveforms. Challenging if multiple things are calling at once.

——

John Pickering, Discover Life

Discover Life – Networking Study Sites to Predict the Impact of Climate Change and Other Factors on Species and Their Interactions

Discover Life – big site, lots of traffic, sunsetting plan is to have project adopted by federal agency or NGO. Mothing project started by Sam Droege, they take photos of moths early in the morning. Interested in climate effects on moths, indicators of air quality, etc. Go outside to porch light at 4 AM, photograph every moth you see, upload to the site, document where you took them, then the crowd comes in and IDs them. About 500 people take moth photos regularly and upload them, but asking them to do it at 4 AM every day takes a special person. These data could be collected by student teams.

Great story about student who saw a rare ladybug and became local expert on moths in county through internship, was uninterested in nature and now on a STEM career path. Reporting data requires taking photo of GIS/camera timestamp and one of cell phone to check time offset errors. Takes photos of the moths with rulers, also frogs if they’re around, and then volunteers start assigning names. ID’ing moths starts w/ location and time of year (much like birds) to narrow down the number of species.

Series of tools to do identification – progression albums, mapper, customized guides. Start w/ shape at rest, further reduces the search space, etc. Very simple series of characters lets you narrow it down pretty quickly, but some are horrible to identify, so they just don’t even try with those.

Can also do mashups with other data sources, harvested on the fly and no local caching. Happy Moths game, going mobile with optimized mobile browser version rather than app, done in HTML5 to get location (etc) automatically. So far have about 120K images, 90% ID’d to species, 5% to genus, other 5% gotta deal with later.

——

Isla Young, Maui Economic Development Board

Women in Technology/STEMworks- Mobilizing K-12 Student Scientists

Hawaii is a very expensive place to live, so many people have multiple jobs and most students have parents working a lot. Goal is to develop a stronger economy, especially in tech, to make it a more sustainable economy for residents. There are a few high-tech companies and she has to fly between islands to visit.

Living wage in Maui County would need to earn over $50K a year for bare minimum survival for a single mom and one or two kids. High tech company people who make a lot more money are there. Cashiers make only about $20K/year. All the talent is being imported and they neither reflect nor are invested in their community.

Working to develop a home grown workforce as key to growth, focusing on women in technology program launched in 2000. Goal is encouraging girls/women, Native Hawaiians and other underrepresented groups to pursue STEM education and careers – which means pretty much everybody on the islands, because there’s lots of cross-breeding so to speak. Want to build resident technical workforce in the state as a pipeline, starting in elementary school. Works with relationships with high tech companies to help students get internships and jobs. About 21K students/year in the program, about 450 teachers, summer camps, mentoring, afterschool programs, much more – very broad set of approaches to getting people involved. Found that kids are much faster to get the technology than teachers, so they teach teachers to just get out of the way and let the students figure it out.

All activities align science with culture. STEMworks program is first STEM/service-learning program in Hawaii, project and service based, software training from industry professionals with advanced tech tools, high tech industry connections, and hands-on, real-world internships. They work with GIS, CAD, game design, web design, digital media, 3D viz, cybersecurity. So far involving 17 schools statewide and 1K middle and high school students.

Key elements of STEMworks is the active involvement, community values and stewardship, they get to use high-end software when they often don’t have computers at home, working to create critical thinkers who can be self-directed learners, and they learn to collaborate in teams with a technical advisor/mentor from their community. Teachers are not teachers, they are facilitators, help guide students to find their own solutions. Already generating cool projects, citizen science focus is an exciting shift as they move forward. They have to ID the problem/opportunity in their community, design project, test solution, develop partnerships, deliver on their project, and maintain the partnership – all key skills for getting students into the workforce.

Works with ESRI for state-wide licenses to manage access to geotech tools, Google Sketchup as local authority – over 215 schools requested the GIS software and over 200 teachers trained in it. Have a STEM conference every year – with an astronomy star party, software competitions, program showcase. One of the projects is “Island Energy Inquiry”, realtime clean energy monitoring that has curriculum to go with it for classroom use, partnered with energy companies to get this going.

Citizen science in HI: only single district school system in the country! Over 400 schools and 170K students throughout state, going to use trust network and partnerships, connecting scientists and educators, create relevant and meaningful experiences through civic engagement at a young age, along with mentoring and leadership skill building. The standardized testing requirements don’t focus on science, the teachers only have to spend 1/2 hr/wk on science so if they’re uncomfortable with it they minimize science in the classroom. Seeing citizen science as an accessible way to get them into science in a different way that’s more approachable.

Starting w/ Hawaii Cricket, Coqui and Gecko Crawl – species relevant to their community. Worked on initial planning and teacher workshop, partnerships w/ USGS and several other groups. Integrating tools including smartphones, iPad, computers, GPS, and using social media, students in their programs do have access to some of these tools.

Trip Report: USGS CDI Citizen Science workshop, day 1

9/11/12 USGS Community Data Integration citizen science workshop

Ice Core Lab, Federal Center, Denver, CO

——

Welcome

Randy Updike – Rocky Mountain Region

Hazard scientist and citizen participation has been critical in his work. Led a NAS team to a volcano in S. America where 25K people had been killed, sheer chaos and most infrastructure was destroyed. Relied upon the locals who had survived to provide local support and tell them what to go see. Citizenry were most valuable resource – had observed the event and knew which features of it were unusual and remarkable, so they lead the team to places where there were interesting things to see. Great trip, not due to the scientists, but rather because of the involvement of the local people.

Has needed to engage the public in this work, and in the past the technology wasn’t there to make it happen, but today it is. Excited to learn how technologies can support the science.

——

Linda Gunderson via WebEx – Director, Office of Science Quality and Integrity

As crowdsourcing and citizen science become more accepted practices, this is really changing the way people do work in exciting ways. In last 2 months, got a call from the White House and learned that Teachers Without Borders were interested in working with USGS on hazards project focused on introducing women and girls to science. It was a phenomenal event. Question from TWB – what would you do if you could harness the passion and attention of 100K women and girls across the globe? What an amazing idea! What they proposed to the White House is to have 100K women and girls, teach them a science curriculum, then have them do vulnerability assessments of their community to incorporate into the global earthquake model.

What would you do with the millions of students and everyday citizens who are interested in our world across the US? How could they contribute to the USGS mission? Need to be careful about data quality and rules, but don’t let those things bog you down in the face of the big questions.

Citizen science is a named objective in coming strategy for DOI. They are interested and want it to happen. Part of the reason is because it’s an ingenious way to harness that power, but also develops scientific literacy, inviting students into a career in science, which is a desperate need for long-term growth, innovation, and the economy. Competition for development of citizen science app, working with citizen science groups, have money to put behind it and it will be announced soon.

——

Jennifer Shirk, Cornell Lab of Ornithology

Exploring the Landscape of Citizen Science and PPSR

Exciting that USGS is exploring identity and role in field of practice. Given the growing landscape of citizen science, how might organizations work together to map out and manage the terrain. We’re exploring this landscape together.

Citizen science is part of the mission statement for CLO; she manages Citizen Science Central, extending involvement in citizen science far beyond birds. Goal of website is sharing best practices and ideas among the community; we need to learn from practitioners and people involved in the diverse kinds of work like citizen science where the public are involved in scientific work. With the growth of citizen science, there’s need for more support for the field.

Sharing brief stories to illustrate the possibilities of citizen science. Terry Root, who’s used data from the Breeding Bird Survey, owes her career to citizen science, documenting bird distribution trends, which wouldn’t be possible without distributed observers. Close to 500 papers have been published with BBS data. She’s also interested in influencing public literacy; how do you do this when you’re working remotely with people you’ll never meet? Terry says she does that by encouraging her students not to hesitate to use citizen science data, with the caveat that you have to ground truth them and determine what data are usable.

Julie Parrish is at UW, where she started COASST to explore status of seabird populations throughout PNW, looking at dead birds on the beach. Designed rigorous protocol for identifying the birds, but also verification with photos and other data. Has been able to understand, publish, and influence management around beached birds to watch for unnatural events. Initially approached Audubon to recruit volunteers, and found birders weren’t a good audience. Needed people who were more interested in the beaches and health of beach ecosystems. She says citizen science is not just about science, but about people doing science about something they love. Learned that you can’t get people to monitor most “important” beach, but need to let them connect to what matters to them, find and celebrate their connection to the place/organism, and that improves retention.

Grupo Tortuguero de las Californias – story of Wallace J Nichols who started an EarthWatch project in Baja to look at populations of highly endangered black seas turtle. Started those expeditions as PhD student because he couldn’t fund the work due to skepticism about even finding these turtles. Partnered not only with EarthWatch but also fisherman and turtle hunters, people whose culture and identity is really linked to the sea turtles. Growing a network throughout Baja and beyond of people supporting research and management that has made a big difference in the recovery of this endangered animal. Who needs this data? Beyond scientists, the locals need and appreciate it. Wrapping up the stories with practical advice: starting these projects, don’t be daunted, basically sold everyone they knew a T-shirt to initially fund the project. Highlight for her at PPSR conference was these people meeting one another.

Reflect on where USGS is in this landscape – by holding workshop and asking questions, USGS is pretty centrally located and can help us come to a better understanding of this terrain in PPSR. Going back to analogy of landscape, how can we collectively map out and navigate the terrain? Lots of work needed just to understand this: what other projects are out there? Who is in my area or organization that I can learn from? What’s already known about how to do it well? Where do I find ideas, resources, papers? What contribution can my project make? These are questions we’re hoping to answer in part through citizenscience.org, will be launching a project database this fall to be a centralized resource for project leaders to curate info about projects & pubs in one place.

Lots of procedural things to consider for getting from inputs to outputs. Power of citizen science is definitely in bulk of observation, and also in experiences – not just deeply meaningful momentary experiences, but experiences that let people know something scientifically and more interestingly. Your work in citizen science will ultimately influence both, but matters in how you achieve these things. Just like good science & good education, science literacy needs to be approached thoughtfully, intentionally, and deliberately.

———

Citizen science policies and challenges session

Introduced by Annie Simpson

Goal is clarifying USGS & govt policies, provide platform for citizen science researchers so that they can share instances of where policy hasn’t worked well for them – opportunities for lessons learned. Want to document how to work better within policy limitations and structures.

——

Lorna Schmid, USGS Infrastructure & Operations Team Lead

Policy Issues with Citizen Science App Development

Why do we need a mobile framework? Addressing multiple challenges and establish support tools and processes – all of the reasons we all know – unknown, limited resources or support for collaboration, funding, data sharing. Goal of project is to develop one stop shop that enables workflow processes, code repositories, mobile community, and common training.

Workshop outcomes on mobile: understanding mobile ecosystem – hardware, app development & delivery. Initial focus on app development life cycle. The process of mobile produce development – 4 phases starting w/ ideation and moving through to delivery. Ideation process includes steps for peer review to ensure that it’s not duplicative and has all the needed parts and will reflect well on USGS. Fairly technical process workflow.

Development workflow: check policy issues, develop communications and marketing plan, mobile community peer review, science review, plus security checklist and specific checklist for development document. Then review/approval for external products (e.g. created by a partner?) involves user testing, security approval, policy review, cost center approval. Once the app is published, need to monitor its performance – if it has low ratings in Apple Store, need to review the product, which reflects on the organization. Issues like scope changes – minor typos, major changes go back to development process.

Recap: training a mobile community, need this to vet applications. Recognize audience dependence and need for phased workflow with documentation. Next steps are a FY13 proposal to continuing developing framework, built out mobile community site, host town halls, continue working with team and advocate for a stronger community.

——

Rob Thieler, USGS Research Scientist w/ Coastal Marine Program

Researcher Point of View: iPlover: A Smartphone Application to Characterize Piping Plover Nest Locations

Found that research scientists collected data in field notebooks but most had smartphones – why not collect right on the phone? Learned several lessons from this.

Business case: sea level rising, this is a very big deal. What to do about it? Natural and cultural resources at risk – produced preliminary map of vulnerable locations. Need to inform decisions about climate change w/ uncertainty management framework; scientific priorities and practices need to change to make it happen.

Looked for poster child, picked Piping Plovers to work with USFWS and NPS, this is his cute charismatic species slide. Piping plovers prefer hazardous beaches for nesting, often subject to storms and other issues. The plovers are hard to see on the beach, nests are even harder to see but are out in the open. Got involved to support decision-making for DOI agencies; listed species, DOI mgt responsibility, interesting and specific habitat requirements that can be predicted, can then feed that back into population dynamics models. Their approach is looking at cascade of info through Bayesian Networks.

Geologist’s view of plovers’ thinking, trying to figure out how to gather data (give them a widget, not a widgeon!) Deploying trained observed and providing protocols, vastly increasing spatial domain, dealing with fuzzy observations, and already know what to collect – screens of app interfaces.

Lessons learned: HTML5 + JavaScript works best. Most devices have issues – wide variations in baseband chips, browser support uneven. Just because someone has a smartphone doesn’t mean they know how to use it! But valuable data can be collected. More lessons: little guidance on mobile apps – Privacy, PII, iconography, branding, OpenID (authentication failure for offline data collection), OMB approval.

24+ organizations monitoring plovers; USGS needs a strategic plan for mobile apps for many reasons. Project needs data to drive research & facilitate decision support. Got a smartphone app working but hurdles included lack of guidance and policy impediments. Demonstrated great opportunities and challenges.

——

Cheryl Smith, USGS Volunteer Program Coordinator (WebEx)

USGS Volunteer Program and Handbook

Views here are from official USGS volunteer management guidelines. HR oversees volunteers but maintains the handbook, is contact for DOI, and give employees access to vol website to post opportunities. Also fields calls and answers questions about vol programs.

Care and feeding of volunteers is done by science centers who work w/ volunteers – the fun part. Volunteers have to track time spent and turn in hours for evaluation. Dept is encouraging more bureaus to develop more vol opportunities due to budget constraints and major contributions of volunteers to USGS. Volunteers have to pass fingerprint check, be US citizens, or have a work visa. That just gives access to the building. Unless they need IT or special access to buildings, that’s the only security requirement for working less than 180 days.

Need to sign special agreement to be a volunteer, no minimum age requirement, but minors need emergency care for minors medical release. Hazard review must be conducted if minors will be involved. No underwater diving or use of firearms.

Volunteer for Science Handbook includes sections on health and safety, related to hazardous conditions, e.g. in labs or boats/watercraft, unscheduled airplanes – this makes gov’t liable for claims against USGS. Main reason to have volunteer properly signed up w/ written agreements, detailed description of their volunteer job, etc. Also includes ethics rules, volunteers have to abide by USGS ethics, e.g. not sharing sensitive data. Constraints in place for all volunteers, scientists emeriti are also part of this scene, become volunteers subject to these constraints, even as former employers.

Post volunteer opportunities on volunteer.gov.

——

Eric Wolf and Barbara Poore, USGS Center for Coastal and Watershed Studies

Researcher Point of View: How are USGS Citizen Projects Impacted by Government Policies?

Current policy issues – Privacy Act (1974), Paperwork Reduction Act (1995), Data Quality Act (2002) fundamental science practices, USGS Volunteer Handbook (2011). Open Government Directive in 2009 is a bit less restricted than USGS Vol Handbook.

Paperwork Reduction Act – supposed to reduce duplicative work (part of the point of some citizen science). Volunteer policy about fingerprinting and long forms really imposes a burden on volunteers. Activities addressed in the handbook give a lot of guidance. Doesn’t cover the lightweight stuff – Acadia Learning, MLMP – do teachers stand in for parents with students groups (in loco parentis)?

Citizen science is global, but people who aren’t citizens or legal aliens can’t be involved. No one outside of US can participate, and there are no guidelines for how to work with these folks. Few guidelines on mobile apps, as we saw. Also mining data from social media that are publicly shared. Concerned about potential for ethically questionable activities in data mining, even if following the law – does government have different responsibilities than corporations?

Shape of citizen science today: who asks questions? Experts. Geography of questions – local, primarily due to policies; participants also local. Knowledge/skills require being fairly educated. Consent must be documented. Research design done by experts. Geography of technology is stationary. Usability is complex (i.e. sucks). Data use is pretty much only for experts, are people collecting the data able to get the data? Basic ethos is that data are shared and are public domain.

In the future, this balance could change. Citizens could ask questions, geography of questions and participants could be global, no knowledge/skills required; unconscious consent; citizens involved in research design; portable technologies; simple usability – no one complains about something being too easy to use; community data use, let everyone explore the data.

——

Shari Baloch and David Newman – USGS Office of Enterprise Information (WebEx)

Lightning Talk: Freedom of Information Act – Employee Responsibilities GIP 140

3 Laws impacting science activities – PWA, Privacy Act, FOIA

Paperwork Reduction Act: requires OMB approval for structured info from 10+ members of the public annually – e.g. a form, web page survey, focus group, phone survey, mobile apps. Public is basically anyone not affiliated with a federal agencies. Point is minimizing burden on public, improve quality and use of federal info. Process takes 6-9 months with 2 public comment periods, so contact them early. If you know of existing data collection without OMB approval, contact them to figure out what the options are for getting approval.

FOIA – most records are subject to it, and USGS promotes transparency – so don’t badmouth anyone in your documents because it could be requested. Some records might be withheld through exemptions. Employee responsibilities: request FOIA rep to coordinate response because of 20-day time limit which is subject to litigation and that’s happened before. Also need to follow steps for FOIA requests (GIP 140).

Privacy Act – responsibility for Personally Identifiable Info – PII. Need to deal w/ privacy act system of records (SORN) so anything that uses unique identifier needs approval from Congress and the DOI Secretary. Suspected breaches need to be reported w/in an hour!

——

Paul Earle, USGS Hazards, for Sophia Liu

Researcher Point of View: USGS Tweet Earthquake Dispatch (@USGSted): Using Twitter for Earthquake Detection and Characterization

Intro slides w/ XKCD comic! Geospatial crowdsourcing – typology includes types of crowds: survivors, diaspora, social net, expert, col; types of sourcing: tagging, mapping, curating, feeding, harvesting.

TED started in 2009 after major earthquake that killed 60-70K people. USGS was criticized, people knew about it on Twitter before USGS did, it got some press. Any seismologist knows it happened but can’t estimate characteristics well enough, so that’s well enough known by the public. Question was how to make that info work for USGS – couldn’t even access Twitter from government computer!

Worked w/ USGS social media group to figure out how to get access to Twitter and do some initial test cases. Found potential ways to use it – just detection, like bloggers said – could also use to get rough location because most tweets have location metadata, sometimes GPS lat-long for about 2%, also have city names, and getting down to the city level is pretty good for earthquakes. Speed of light travels faster than the speed of sound, so using Twitter is faster than seismic stations, which are pretty sparse globally. E.g., in China, the seismographic data is delayed by 15 minutes before it’s sent to USGS, so they had no info about that huge earthquake in 2009.

Got framework set up by a computer science student, who then got a job – so they got another student, but there’s a lot of variability in student motivation and skills. After that person, got a little more money and got some professionals in on it to build a robust system to gather tweets and send out public messages about seismically verified events. That framework provided ability to build other applications off of it. Product has been rock solid, big advantage of using professional developers. One of few systems that works better in real time than on static data!

Performance: has had detections in as little as 18 seconds! Average of 3 events per day, 95% correlated with true seismic activity. Faster than $200M seismic tracking system. Alert delay is 26 – 237 seconds. Magnitude of detected events from 2.5 – 7.4 on Richter Scale. Even though it requires jumping through hoops, it really does what it’s supposed to do, there’s no way to do this as an individual researcher.

——

Kevin Gallagher – Associate Director, Core Science Systems

Lots of discussion about engagement driven about new technology, but citizen science has been going on very successfully for a very long time, particularly conservation efforts. So what’s really new here? The opportunity has been here for a long time, makes mission more relevant. Competing at national level for funding with every other agency. Opportunity to engage w/ citizens lets USGS showcase assets and work, and improve mission and relevancy in public eye.

What’s really new is the tech that lets us engage in new ways, e.g. mobile devices. Need diverse talents to realize the potential, how to simplify and move forward. Policy is a big hurdle, so a subcommittee that can work on those issues to help support smoother, faster approvals would be valuable. Building out mobile computing infrastructure also needed, and sharing stories too.

Some discussion of attendees’ goals in being at the workshop – appreciation from “outsiders” for being present and being able to engage, appreciation from USGS for having non-USGS people involved. Need to start getting the house in order; there’s no question that as things move forward, there’s need to work with partners and steal shamelessly from what other agencies are doing. Committed to making this successful, creating forums, investing in staff support, assist in moving policy, put funding on the line to make it happen.

——

Matt Cannister, USGS Biological Technician (Southeastern Ecological Science Center)

Researcher Point of View – An Overview of the USGS Nonindigenous Aquatic Species (NAS) Online Sighting Report System and Lessons Learned from Mobile App Development

75K+ records for 1K+ nonnative freshwater (mostly) species, each record with up to 60+ data fields and public can see 20-30 data fields. Also show point and HUC (watershed?) distribution maps, e.g., for Lionfish. Also does species accounts with fact sheets for public reference. Now have an alert system for Early Detection/Rapid Response aspect – big aspect of invasion biology, get rid of the problem before it’s a problem. Alerts generated when species observed in novel area, goes out through RSS feed and tweeted, also email alerts according to state, group of species, specific species, etc. Have issued almost 1200 alerts, 62% from personal communications through Online Sighting Report form, 83% of confirmed Lionfish are from public reporting. People can fill in various fields, upload images for verification, and lots of disclaimers.

Want to turn that into a mobile app. One constraint with online reporting is follow-up after field time to enter data, potential of losing details. Want to make an app like iPlover but for aquatic invasives. They have the app! Working prototype emerged in 3 months, fully functional with back-end server support. But the issues making it a “sort of” include: no technological issues, no financial issues, problems are all bureaucratic. Have been waiting since November 2011 for DOI to approve Management/Review/Approval process for app development before the app can be sent out to app stores.

——

Policy discussion

TOS for Apple and Google app stores are not acceptable for USGS, though some projects have achieved this one-off, e.g. NPN had one. NPN collected millionth data point, and did emergency data collection approval with OMB, had to jump through many hoops, got scolded, have 6 month approval and now have to go through the rest of those steps. Just getting sign-off on TOS doesn’t mean any of the rest of the stuff is ready to go.

There is now a fast-track for OMB that is actually working, takes a lot less time to use a web form – as little as three weeks for questionnaire approval compared to 2 years. Hard to speak to all the parts of the process, maybe need a checklist of how to go through all of these steps.

Need to avoid hiding USGS work in universities, etc. Already fighting for funding, need credit for this work. Things like mobile app dev framework will help navigate some of those other issues. Having people use universities as an intermediary dilutes credit to USGS. Need for partnerships that meet all players’ needs. Concerns about university partnerships being less stable than USGS needs, e.g., due to funding cycles.

Nothing stopping USGS from developing tools (subject to review process) and releasing as open source, e.g. technologies and protocols which are huge amount of effort to develop. That makes much bigger potential impact.

Need vision statement and direction for moving this forward at USGS level. How much would a general strategic plan help? It would make a difference to establish big questions to answer, how to work together, the kinds of communities to form, and issues that need to be addressed, e.g. policy. It could be a very simple framework, but that always helps.

When were these showstopper policies put in place? Are they outdated? Doing the workarounds could show the value of updating those policies, because that’s the really big roadblock.

What have been major issues with PII in citizen science at USGS? Wouldn’t say mistakes are made, requirements are always understood, but organizers are sometimes doing it their own way rather than going through the official processes. Some misunderstanding of what PII is – just a name isn’t PII, but two different pieces of related, linkable PII is. Lacking a unique identifier makes the data problematic for research use because of data quality concerns.

Policies so complex (and numerous) that there’s room for multiple interpretations.

How can USGS scientists figure out policy and resources to recruit effectively within policy guidelines? Hard to figure out what the full process is, right now it’s word of mouth, don’t know where to go for that info. There’s a new website to try to help document all that info, building a one-stop-shop.

Memo from OSTP – things like 3rd party ID systems embedded in them, but virtually no one seems to know about that memo. Also don’t know when there are updates to policy – keeping people up to date on changes. Collecting up the policy knowledge in the room would help generate a pretty comprehensive list.

Many of these issues are burden to domain researchers, they shouldn’t have to deal with all the policy and other related stuff outside of their areas of expertise and be pulled out of their research. Same issue in data management. Need an ombudsperson or liaison to help researchers navigate these issues.

Sally Holl discovered Code for America, suggests creating Code for Science proposal. Science sector lags behind other areas, and a lot of government data is not available, that which was is not all that usable. Need to be able to host hackathons with staff providing bridges between domain science and IT development. Doing this would require hiring a group rather than individuals who all need to be fingerprinted, etc.

Challenge.gov contest – USGS has never done this before, Abby is working on putting a proposal for a challenge together.

———

Engaging the Public in Scientific Research Session

——

My talk – The Evolving Landscape of Citizen Science

——

Barb Horn – State of Colorado (WebEx) NWQMC

Engaging the Public in Scientific Research Perspectives from the Volunteer Monitoring Community

Volunteers as workforce – different model and set of resources to support; involving volunteers makes it more than just research. Volunteer monitoring community is sort of its own community, especially water quality monitoring. Trying to explain to National Water Monitoring Council, the volunteer monitoring projects do pretty much the same thing as professionals. Data management and technologies are an issue as for everyone else.

Started out as primarily education and stewardship, but has moved toward decision-making and taking action. Planning phase steps really don’t show how to deal with the people part – figuring out who the decision-makers are, what info they need, and that’s what should drive the rest of the planning steps.

Planning process starts with what exists, formulate objectives, how to manage and analyze, how to deliver to decision-maker, evaluation of project success, what needs documentation, and what’s the appropriate network to use those assets. General approaches are tiered – based on amount of rigor and time involved. The advantage of the simple starting point is that it’s a good starting point and you can potentially move people up into a higher tier, but the disadvantages have to do with data interoperability, not the right level of data for the particular location. Other approach is one-size-fits-all, all the resources go into one product/service/deliver/support, supports interoperability, but costs more up front, may lose people who want to do the more rigorous work, and might not meet all the objectives.

River Watch is a one-size-fits-all. Annually they monitor around 700 stations a year, including field, lab analysis, physical habitat, macroinvertebrates, photo. Everyone does same training and if they can’t meet QA/QC cut-off points, they can’t continue to participate. Equipment issues – loaning, giving equipment w/ either the group keeping the equip or they reclaim it, etc. Standardization is required across methodology for this kind of data.

Analysis approaches: some do it all, others collect and hand over to experts, some do a combination of those approaches, but in all cases it has to be standardized. Study design may be very different for different participant groups, and sometimes a group starts with one study design and then change the design to match the outcomes, then maybe do intervention and later go back and re-check. So monitoring can evolve and study design changes. Survey on demographics and motivations; main purposes are passion and making a difference. Should document volunteer responsibility, what sponsors support, and responsibilities – typical volunteer coordination considerations.

——

Greg Newman, Colorado State University

CitSci.org: Cyberinfrastructure Support for Grassroots Conservation, Citizen Science, and Community-based Monitoring

Learn from the masses, and then work with them – the art of teaching is the art of assisting discovery – these are things that make citizen science work. Keys to success: volunteers, scientists, & mavens who make the projects happen; need all three roles to make a project work.

Today we have countless projects, creating volumes of data, and they often focus on pre-defined topics – birds, phenology, etc. What if you want to measure something else? With that question in mind, felt there was a need for a platform to help new projects get going. Got NSF money and built such a system, facilitate creation of projects, encourage cross-project partnerships, support data management. Platform supports both top-down and bottom-up approaches to project design. Steps in project development are far more complex than most people realize.

Simple web portal for setting up projects – personalizing with photos, etc, build their own data sheets, etc. System allows membership gatekeeping to help support data quality, build data sheets from typical template items, add specific attributes and site characteristics. With specifying attributes, they have a growing number of attributes to manage, e.g. species attributes like height, weight, etc. The goal of doing this is standardized, interoperable, and flexible data – integrating data from Pika Watch and Pika Net, all adopting same protocols. They determine what to measure – so far, species attributes and site characteristics. But it could be kW energy use in houses as a new attribute in the system. Their mantra: If you want to measure it, you can measure it, but be sure no one else is already measuring it!

For data analysis, there are automatically updated canned reports, but working to build custom analysis where they can pick their own variables. In the future, hoping to develop open APIs and extend interoperability. There’s also a feedback tool for surveying volunteers. So far they’ve had 40+ projects, 7K+ observations, tech reports, academic publications, etc.

Making online citizen science successful: shared praxis, amplified collective intelligence, restructured expert attention. What motivates people? Sharing ownership, prompt them along the way, reward and respect them. Are rewards effective? Communicating and sharing results is number one – that’s why they’re participating. Show progress, make sure they know they’re making headway toward an answer to the research question. Give attribution – really acknowledge them. Harnessing these motivations: adopt common protocols, encourage standards while remaining flexible. How to succeed? Train the trainer, training a coordinator who trains volunteers, crowdsourcing to distributed groups. Each model of participation has their place in the bigger picture.

So what’s next? Event management, online training support, volunteer tracking, improving project social media marketing, integrate w/ SciStarter to promote projects. Expand analysis, customized reporting, mobile apps, and blogging features.

——

Greg Matthews, USGS National Map Corps

Volunteer Map Data Collection – The National Map Corps

National Map Corps – volunteer geographic info used for improving topo maps, but eventually got shut down. Restarted in 2010 with OSM; started with software – OSM had quite a few advantages, but needed a willing partner, which ended up being State of Kansas data access and support center. Took their data set and cross-walked to USGS best practices model, then loaded into the system so editors could edit roads. Kansas focused on state routes, while USGS focused on interstate routes. Aside from a few hitches, it has worked well.

Guidelines for editing also important – they had the same reasons for doing the editing, which is fairly rare. Successfully deployed the system customized to USGS specs, worked with co-editing using cross-agency shared specs, and used lessons learned for phase 2. This is a large federal agency, but had 3 staff w/ about 1/3 of their time each, so it was a large project.

Phase 2 involved structures (buildings), volunteers from student bodies, evaluating quality/cost to ensure there would be ongoing internal support, also very important to make it available to The National Map and meet all their quality needs. Still successfully using OSM technology stack, and trying to contribute the data back to that community as well. Editing interface was much improved using user-centered design. Contributed points include a number of attributes. Was specifically collecting the info for the structures program. Seeded project area with gazeteer info on structures, shown in red, checked over by editors and then approved or changed, and then shown in yellow, so you don’t get duplication of effort, map itself shows status of editing tasks. Also involved peer review – editors checking each others’ work.

Outreach and communication was easy using Google Sites, so it was unofficial and only available to those who were invited to participate. Forum was also set up in the hopes of reducing volunteer management load as well as collecting feedback. Basic thank-you certificate of achievement, considered it the least they could do, certificate shows how many structures were included on The National Map – pretty popular with some people.

Had over 1100 points contributed or edited, lots of new points added, and some historical points removed. Positional accuracy also improved – from 50% accuracy to 84% accuracy, and then after peer review 92% accuracy without USGS involvement, which met National Map standards of 90% accuracy w/in 40 feet.

Phase 3 – now collecting a new type of data guided by needs of The National Map. Content management in Confluence, with directions, video tutorials, no login and now official with approval. Editing interface is purpose-built to make it as easy as possible to contribute to the system. Adopt-a-Quad system lets people review all the data in a quadrant, more technical job and asks a little more of those people, going well so far. Volunteer contributions after public announcement 3 weeks ago: about 1500 data contributions, almost finished half of Colorado in very short order. Hoping to scale up beyond CO in next few months for these 10 structure types, potentially serving a larger set of purposes beyond USGS.

——

Kristi Wallace, USGS Alaska Volcano Observatory

The Alaska Volcano Observatory Citizen Network Ash Collection and Observation Program

Collecting observational data, no outreach people in this program, so it’s an extension of their work that they’re taking on. Research focus is on volcanic ash, involving volunteers in collecting observations and samples of ash. Over 100 volcanoes and most are distant from Anchorage, so they need distributed contributors with timely access to ash plumes. The data are used to ground truth satellite data and ash fall models, and another important need is met by collaborating with NWS Ashfall Advisories, since they are the agency that releases those advisories in real time. So timeliness is really important. The data points are used for isomass maps and samples let them understand composition of the ash. Aleutian Arc volcanoes require volunteers to collect data.

The ashfall models are not yet public, but that ground truth validation is really needed, so they can be viewed by the public during eruptions. They use a variety of training forums, with written instructions, video, workshops, and targeting key citizen groups. Take a variety of observation and sample types, complex to simple; eyewitness accounts, thickness measurements, measured-area sampling, time incremental sampling, bulk sampling. Give people the choice to report what they feel able to do; provide very explicit and simple instructions for all those activities.

Website includes dashboard for each volcano – intro material, reporting, and send email. They put instructions in a variety of media to make sure everyone can use the format that works best for them. Instructions include lots of pictures to make it very obvious, diagrams, etc – might feel like you’re insulting people, but a lot of people just don’t know how to measure a baking pan’s dimensions. Data sheet includes metric ruler because people use inches when they mean centimeters. They also hold community workshops in downwind communities, great way to get long-term participants who become really loyal and they keep in touch – she calls in advance when precursor activity starts up so they can refresh themselves on their procedures.

Also working with National Weather Service Spotters, who are highly trained, just as excited about volcano science as weather science, and they tend to be very valuable. Lots of observations and samples sent in – about 250 observations and 55 samples were sent in. They send out hand-written thank-you’s with pics of the eruption that they sent a sample for to make sure people know it’s appreciated. Sometimes samples aren’t well documented, but there are some that demonstrate extremely careful documentation of their samples and procedures.

Just added ashfall recording page. Don’t have an outreach person, she has to be in the field during eruptions or have volunteers to do it, very stressful. So they put together an online database, 4-step form based on data that are entered. Time/date, several ways to identify locations, yes/no on whether ash was present, details about the ashfall event, and then if you sampled, more questions on the type of sample and comments. Optional contact info – need to be able to follow up on many of the observations for weird reports or special data collection, allows people to opt out of providing PII. Have a map that shows 24-hour reports of ashfall, shows only yes/no whether there is ash or not, and locations aren’t very specific, but specific enough to be useful for public.

Can’t use this system yet due to bureaucracy, but designed it in a way that is not volcano-specific, so it can be used globally. Gave a copy of software to New Zealand to use – they will probably have it running before the US does. Back end tracking with CRM-like functionality, etc. Image database now available, growing fast, and photographers love that they get credit for it.

PPSR’s Contributions to Science

Conference on Public Participation in Scientific Research, Day 1 Session 2 – 8/4/2012

—–

To Use or Not To Use: Is That the Data?
Terry Root

Examples of PPSR biogeography – huge collections with enormous monetary value. Can go back to the 1800s to understand egg laying for phenology changes. Everything was fine with older data – 800K eggs have been digitized, but newer data is problematic because eggs couldn’t be collected. So now nest record cards need to be digitized. These data were used to save peregrine falcon and brown pelican from DDT risks.

Starting in Victorian times, people were interested in learning more about what they saw but no field guides available – so Arm & Hammer distributed species cards in their baking soda! Has evolved into Christmas Bird Count through a circuitous route.

Once data were computerized, they started to find data could be used very well to answer RQs. Birdwatchers knew more about irruptive phenomena than scientists, probably caused by climate. Was then able to look at distribution and abundance from CBC data and find out the range change for species.

Many other data sets go way back, priceless info saved by hiding under a mattress! Many of the historical data are in private and museum collections. Big growth in large long-term datasets that are badly needed to address big questions. We are now poised on the edge of a huge explosion of data, but what does misidentification mean for data quality? We used to throw away these data, but now we can use smartphones with cameras and social networks to get info about many things from large number of people into scientifically useable datasets. iNaturalist a great example of how this is working really well – already seeing exponential growth and going viral soon. 3 yo can use it and get excited about finding out what a species is.

The Role of “Citizen Science” in Weather and Climate Research
Noel Doesken

Early traditions of weather observation started in US by Ben Franklin and Thomas Jefferson. They communicated to try to understand what was going on, but didn’t have spatial and temporal context. Smithsonian project from 1849 introduced new technologies, telegraph to share weather observations.

Analysis and interpretation of volunteer data more difficult than recruiting volunteers, getting as many as 500K data points/year, which was when standardization became a big issue. Took 12 years to report on the data, so volunteers had to be very patient.

Colorado state weather service started establishing state-based weather observing networks in the late 1880s with only $2K. Within 10 years there was a solid reporting network, led to nationwide “Cooperative Network” that continues today. First purposes were simple – climate resources of the country, particularly what crops could be grown where and when, also equally important to predict extreme weather.

In bigger picture, most of the data are very skewed geographically, but a very impressive foundation. Many applications, both scientific and practical, such as climate and health – “a stinking big deal.” Have learned from historical data that there are weather cycles that is helping model weather. Dustbowl and depression increased interest in weather and climate, advanced use of volunteer data. Majority of drought monitoring is from citizen science.

Naturally this leads to understanding climate change – CoCoRaHS is keeping this going. We need more rain gauges and finer granularity of placement.

Foldit and Games for Scientific Discovery
Seth Cooper

Many people playing video games, what if we could direct that to solving problems for science? Combining human and computational power, as well as a way to motivate people to engage and solve problems they didn’t think they could contribute to.

One area where there’s lots of potential is biochemistry – proteins and protein folding. Very important part of life. Two ways to look at it, sequences and 3D structures. Hard to solve folding problems algorithmically. Foldit lets gamers use the 3D visualizations and both human and computational tools to solve problems. Have had 250K people play the game, over 100 protein structure puzzles.

Scoring and leaderboards help promote competition which motivates gamers. Technical structures are complex, but robust for solving difficult puzzles. Constantly releasing new features and bug fixes, and giving players feedback. Worldwide community participating, multiple languages. Players have produced very exciting results, protein related to AIDS virus in monkeys, algorithms failed but players succeeded in 3 weeks!

Ended up implementing scripting structure for “recipes” so players could reuse functions – player algorithm independently discovered scientist algorithms, perform better! Made trophy for early winning player, keeps it on his desk. Also co-creating structures, now making interface tools for scientists – when tool is fun and easy for everyone, also useful for scientists.

The Many Benefits of PPSR
Linda Silka

CBPR – community based participatory research – no research on us without us. Academic research may not be the right way to address problems.

Working with tribal groups on emerald ash borer in Maine – not many ash trees but they are critical to tribal traditions and economic opportunities. CBPR is growing just like citizen science – there are organizations, journals, and grants for training and cross-disciplinary support.

CBPR successes – adding rigor to data collection, need to merge professional and local knowledge to solve problems. Example from tribal lands in Nevada about nuclear contamination – researcher vector model didn’t take into community food sources. Other examples of ways that community knowledge is strengthening scientific outcomes, e.g. incinerators and air quality household health studies – very concerned about children but had concerns, wanted dialogue with researchers. Similar outcomes in emerald ash borer studies, nutritional studies.

Linking knowledge to action by bringing in local stakeholders; federal research agencies/foundations reviewing proposals differently to promote broader impacts. Using research cycle as tool to understand issues that emerge at each stage. Many questions remain about assumptions and unknowns.

Lots of resources at http://CCPH.info.