Archive for April, 2008

Loretta Auvil was invited to present the keynote address at the Text Mining Workshop 2008, which was held in conjunction with the Eighth SIAM International Conference on Data Mining (SDM 2008) in Atlanta, GA on April 26, 2008.  Her presentation title echoes SEASR’s identifying phrase, “Engineering Knowledge for the Humanities.”



Over the last decade NCSA’s Automated Learning Group has innovated data mining technologies for industry, government, and the sciences. In the past few years, we have broadened our focus to include knowledge discovery in the humanities. My presentation will focus on how we are negotiating humanities computing’s special challenges for data mining and analysis. I will discuss our early collaborative projects, FeatureLens and Nora, and SEASR (Software Environment for the Advancement of Scholarly Research), the Andrew W. Mellon Foundation-funded project we are now leading. Each of these projects has developed technologies customized to meet specific needs of the digital humanities community. FeatureLens–an early MONK (Metadata Offer New Knowledge) application–uses the machine learning approach of frequent pattern mining to identify fuzzy repetition patterns in a data collection, and with no initial human input. Nora–a case study for eighteenth- and nineteenth-century British and American literature–uses predictive modeling techniques to classify documents, even given complex and notoriously indistinct expert classes such as sentimental fiction. SEASR is our most ambitious project yet, employing a semantic-based, service-oriented architecture to build software bridges that allow users to access data stored in disparate formats and on incompatible platforms and to provide an enhanced environment for workflow and data sharing. The essential infrastructure SEASR provides will advance the capabilities of projects like our partner, MONK, a digital environment designed to help humanities scholars discover and analyze patterns.

The SEASR and NEMA (Networked Environment for Music Analysis) teams have transformed a dynamic music classification explorer developed by IMIRSEL (The International Music Information Retrieval Systems Evaluation Laboratory) into a SEASR application that can be reused in whole or part by music researchers everywhere. Ira Fuchs–Vice President of Research in Information Technology for The Andrew W. Mellon Foundation (sponsor of SEASR and NEMA)–gave the “Son of Blinkie” (SoB) explorer its first demonstration on April 16th.


Innovations in digital technologies have changed the ways we create, access, analyze, share, and consume information. But to realize their full potential, we need to re-evaluate digital information technologies to consider whether their methods are hold-outs from the age of print and, if so, what improved means we can devise. IMIRSEL’s SoB [1, 2], a dynamic classification explorer for musical digital library users and researchers, offers such an advance to the way in which we access and analyze music.

In the print collections and their digital descendents, information is retrieved through metadata, or descriptive labels, imposed upon it by librarians, editors, and domain experts. This metadata is used to generate tables of contents, subject indexes, and other searchable formats. Once determined, such labels and their associated epistemologies tend to become fixed and accepted as fact; they present a closed system of established knowledge rather than provide a virtual landscape that encourages exploration and enables discovery.In developing Son of Blinkie—affectionately named after the earlier, simpler “Blinkie Thing” [3]—the researchers at IMIRSEL have sought to bring leading machine learning methods to bear on the problem of how to make better use of the now digital nature of music collections. They have developed a means for searching music automatically, using its features of composition rather than imposed metadata as a guide. Not only does this automated method improve the speed and accuracy of information retrieval, but it promises to enrich our understanding of music and its classification.

Faced with a collection of music, we often accept that the labels imposed by past listeners are accurate and/or informative. But listeners may hold conflicting opinions about a piece, and the piece itself may defy reductive labeling. Through analyzing a piece using its own compositional features, machine learning can help us to understand whether a given piece is representative of a genre or mood as a whole or to certain compositional tendencies within it, tendencies that may change over time, by performer, or even by performance. What’s more, Son of Blinkie (SoB) advances earlier attempts to automate digital music collection retrieval and analysis.

Consider the traditional train-test approach to building, evaluating, and using machine-generated audio-based classifications (e.g., genre, mood, artist, etc.) for Music Digital Libraries (MDL). It’s useful in some contexts, but has two serious shortcomings. First, the classifications are monic (i.e., only one class label per piece). This monicity ignores the fact that most music comprises a mix of moods and/or genres, etc. Second, the classifications are static (i.e., one class label per song) even though pieces evolve through several moods and/or genre mixes over their play time. The SoB system offers a new and superior method of digital music exploration, engineered to overcome train-test shortcomings and better capture the dynamic nature of music. SoB provides users with the capacity for highly configurable real-time classification, visualization, and audition.

Another important advancement made with SoB is that the application operates within SEASR’s service-oriented architecture, taking the form of a series of reusable, open-source components managed by and executed as a shareable workflow from SEASR’s community hub. Not only can users run SoB against their own data sets– with SEASR’s assistance in accepting different input formats stored on different platforms–but they can also reuse and revise components and workflows to build their own music research applications.


SoB works by extracting a stream of features from audio tracks and applying a set of pre-trained classification models to short windows (10 sec.) of these features to generate posterior probability distributions in real-time. The display of the classification probabilities is synchronized with the audio playback, empowering users to dynamically explore the effects and interactions of an infinite number of parameters involved in automatic music classification. SoB permits users to select an arbitrary number of classification models from the system’s ever-growing model library. Currently SoB’s model library comprises two classification “task” collections: mood and genre classifiers.


Above, we show a user simultaneously exploring the different real-time behaviors of mood classification models and genre classification models. Each model is making different predictions on this particular 5-second slice of the incoming, never-heard-before, song. The user can visualize the models’ prediction probability distributions, which can help the user better appreciate the potential “mixture” of moods present. The user can also listen to the synchronized audio to better understand the strengths/weaknesses of each model.

Below is a view that shows how data flows through the Son of Blinkie system, as it operates within SEASR (specifically, the semantic, web-driven dataflow execution environment portion of SEASR, which we have named Meandre). Each component represents one step in processing the data. The components run (and so process data) in the order established by the flow: from receiving the song filename and model filenames from the web application, to loading the audio and model data into memory, to extracting a variety of features from the song, to applying the model to the extracted features, to returning the predicted results to the SEASR community hub (a web application) for visualization. Every time a different song is selected, the web application executes this same flow.



  1. Funded by The Andrew W. Mellon Foundation and the National Science Foundation (Grant No. NSF IIS-0327371). Thanks to M. C. Jones and the SEASR team for their technical assistance.
  2. IMIRSEL is directed by Dr. J. Stephen Downie, Graduate School of Library and Information Science (GSLIS), UIUC ( His Co-PIs on the Son of Blinkie system are Kris West, School of Computing Science, University of East Anglia and Xiao Hu, GSLIS, UIUC.
  3. Downie, J.S., Ehmann, A.F., and Tcheng, D. 2005: Real-time genre classification for music digital libraries. JCDL’05, 337.
  4. NEMA Website:
  5. SEASR Website:

Throughout March, SEASR and I-CHASS hosted humanities and social sciences research teams selected for their diversity of approach and interest:

Global Middle Ages: “Global middle ages” is a term in Medieval Studies that designates an interest in the middle ages across the world, i.e., non-Western societies.  The research group (Susan Noakes, French and Italian, Medieval Period, U. Minnesota; Geraldine Heng, Medieval and Women’s Studies, English, U. Texas-Austin; Ayhan Aytes, doctoral candidate in Communication at UC-San Diego, and also a medievalist) thus intends to create a digital resource that establishes and enriches researchers’ understanding of how non-Western societies contributed to medieval European culture (approximately 500-1450 ce).  The design for this project is centered on a mapped narrative of cultural influences coming out of Africa (e.g., the former provinces of Rome in the north, including Egypt; later, Islamicized Africa, especially Moorish civilization; and, later still, Western Africa as a site of empires as well as the transatlantic slave trade).  It will thus ground the historical for users through appeals to their temporal, visual, and spatial imaginations.  As with digital timelines, such mapped narratives tend to offer waypoints to users at which they can “stop” to browse in-depth information provided in a variety of media forms.

Peace and Nonviolence:  This project brings together researchers who have worked to promote peace and non-violence through informed activism.  They are uniformly interested in the social causes of violence.  Steven Valdivia, Independent Scholar (former Executive Director, Crisis Intervention Network-LA), and Fernando Hernandez, Education, CSU-Los Angeles (emeritus) are two researchers working on LA gangs.  They are especially interested in how governmental responses to poverty, minority status, and gang activity have fostered gang formation and violence.  They are seeking means of counteracting gang formation that might be recommended as public policies.  One theory they hope to prove is that the militarization of response to gang activity has worsened rather than improved gang violence.  The researchers from the Southern Poverty Law Center (Mark Potok and Heidi Beirich) are interested in research subjects that fit their civil rights mission, which the center pursues through its “tolerance education programs, its legal victories against white supremacists and its tracking of hate groups.”  They are especially concerned with researching the formation of hate groups (e.g. white supremacist), particularly how they hail new members.

Digital Portfolio Project:  Virginia Kuhn (Research Assistant Professor, Associate Director of the Institute of Multimedia Literacy, USC School of Cinematic Arts) has just led the first class through a new, intensive program at IML.  Their senior year culminates in a major multimedia design project, producing a finished piece with support work for each of the 30+ students.  Because archiving technology is increasingly available and because the new program is an important focus for the school, Dr. Kuhn wants to find a stable and innovative means for archiving these projects and retrieving information from them—with her ultimate goal being to produce a persistent, state-of-the-art pedagogical resource at USC, one that could serve as a model for other programs.  According to Dr. Kuhn’s official faculty bio, the “project was recently awarded a large (3 terabyte) allowance of storage space on SDSC’s TeraGrid.”  Consulting on the project are ISU’s Cheryl Ball, a specialist in digital composition and rhetoric (English) and Editor of Kairos, and Elijah Wright, a doctoral candidate at Indiana University’s School of Library and Information Science.

We are working with these teams to apply and further develop SEASR’s capabilities, and will feature their projects in a SEASR community-building workshop later this summer.

Last week, members of the SEASR team attended the Hadoop Summit & Big Data Computing Study Group. Bernie Acs and Xavier Llorà share the following report of the meeting:

The two events hosted by Yahoo! and sponsored by the Community Computer Consortium shared the common theme of data intensive computing. The first focused on the current state of Hadoop[1], which is an open source Java implementation of Google’s MapReduce [2] programming model and a supporting infrastructure that includes a distributed file system (HDFS). Based on GFS3 [3], HDFS is highly distributable and fault-tolerant. The second event was a symposium that featured several talks given by academic leaders and industry representatives who presented their work and perspectives on the types of computing resources that will be necessary to address the increases in data volumes that will be encountered.

A pervasive idea carried through both sessions was that significant internet companies like Google, Yahoo!, Amazon, and Microsoft are driving forward innovations and resources that will likely force new paradigms for how large computational environments should or could be developed and utilized. Below, we note the influence these companies are having on HPC:

* Google is making available to the educational community an HPC cluster that will be managed by IBM and allocated by NSF Christophe Bisciglia of Google was present at both sessions to serve as a speaker’s guest and to introduce this partnership).

* Yahoo! has implemented the largest production of a Hadoop infrastructure and has provided as open source the contributions developed at Yahoo! Research to enhance it. One is ZooKeeper [4], a highly available and reliable coordination system. Another is the Pig [5] scripting package.

* Amazon’s Elastic Computing Cloud (EC2) [6] service offering is providing virtual machine instances and virtualized clusters ready to run Hadoop applications.

* Microsoft Research is developing an infrastructure called “Dryad” [7], which is built using .NET framework, SQL Server 2005, and SQL Server Integration Services (SSIS) [8]. Dryad offers a highly scalable computing environment with fault-tolerance, scheduling, and job management.

Data-intensive computing software architecture paradigms are shifting rapidly with the concepts of cloud computing and the virtualization of computing nodes or even the virtualization of entire computing clusters. These factors are influencing the future of HPC computing concepts and will potentially help to define new areas of focus and gain traction with a variety research communities suffering from data overflow, like biology, genomics, astronomy, and other sciences. Hadoop Summit (March 25th, 2008).

Approximately 400 attendees were present to hear about the current status of the Hadoop infrastructure project, learn about the future directions of the project, and see showcased some real world applications that use Hadoop. A detailed agenda of speakers can be found here. Some of the most interesting application implementation talks were those presented by Kevin Beyer (IBM) on JAQL JSON Query Language, Andy Konwinski (UC Berkely) on X-trace debugger, Steve Schlosser (Intel) on ground modeling application, Mike Haley (Autodesk) on object library processing, and Dr. Jimmy Lin (U. Maryland) on a cluster computing course for natural language processing. Big Data Computing Study Group (March 26th, 2008).

The symposium was well attended by many distinguished participants. Speakers presented some impressive examples of large data research focuses, which are swimming in oceans of ever expanding data. Two presenters, Dr. Jill Mesirov (MIT &Harvard) and Dr. Alex Szalay (John Hopkins), talked about their respective science areas, how these fields are evolving with new technology, and how that is presenting new issues for managing exponential increases in data volumes. Jeannette Wing (NSF) introduced the Google/IBM/NSF partnership as a new and welcome innovation initiated by industry. Dr. Randy Bryant closed the session with the sentiment that services like those from Amazon EC2/S3 could lead to academia’s rethinking HPC ownership. A complete speaker agenda along with abstracts can be found here and detailed summaries of the talks can be found here.


[1] Hadoop infrastructure project pages.

[2] Google’s MapReduce was presented during OSDI’04: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, December, 2004. Paper and Slides by Jeffrey Dean and Sanjay Ghemawat available here.

[3] “Google and the Wisdom of Clouds” article dated December 13, 2007, printed in the December 24, 2007 issue of Business Week subtitled “Google’s Next Big Dream”; article viewable here.

[4] Distributed applications use ZooKeeper to store and mediate updates for key configuration information and can be used for leader election, group membership, configuration maintenance, etc. Additional information availablehere.

[5] Pig is a language for processing data and a set of evaluation mechanisms for local execution or translation into a series of map reduction operations for execution by a Hadoop cluster. Additional information available here.

[6] Highlight page, service description, and pricing can be viewed here.

[7] Dryad handles job creation and management, resource management, job monitoring and visualization, fault tolerance, re-execution, scheduling, and accounting. Additional information and links are available at here.

[8] SQL Server Integration Services in SQL Server 2005 provides a scripting capability that allows developers to create complex data transformations, data mining, and manipulations using graphical interfaces and/or .NET scripting. Additional information is available here.