Methods and Data

As promised, after some initial posts introducing the project and displaying some results, the next few posts will concern data and methods.

PROJECT METHODS AND DATA

Some of the methodological choices I have made for this project derive from the terms of the Digital Innovation Fellowship from the American Council of Learned Societies: for example, the choice to disseminate the products as broadly and freely as possible via the Web and to use the project as an opportunity to encourage and teach others how to undertake other such digital humanities innovations. Other choices relate to funding limitations, such as the choice to use this free blog to disseminate the results because the fellowship mainly provides salary replacement for one academic year rather than research funds to acquire expensive equipment such as a server or pay for ongoing costs such as annual fees. Some of the choices also align with personal values, like the one to use as much free software as possible, such as the ArcGIS Online map viewer, in order to allow anyone with a broadband connection to use and modify the GIS. And yet other choices emerged from the research process itself: for example, my fellowship proposal focused on Atlantic commodity networks in the nineteenth century, but the scope of the project soon expanded to all oceans and the seventeenth century through the nineteenth because once I actually began I realized that producing a worldwide GIS for the 1600s through the 1800s would require only a little more effort and avoid conceptually and technically problematic compromises such as excluding voyages that entered the Atlantic from the Indian Ocean or that began in 1799 but ended in 1801.

Software and Hardware

The principal software packages I used were either free web applications or ones for which LSU had existing site licenses. The licensed ones included Microsoft Excel, Word, and Access 2010; Adobe Photoshop and Illustrator CS6; ArcGIS Server; and ArcMap and ArcCatalog 10. The free ones included the non-subscription version of ArcGIS Online and GoogleBlogger, Maps, Earth, Maps Engine, Sites, and Translate. The free version of ArcGIS Online does not allow the addition of time-enable layers, raster layers, or vector layers with more than 1,000 features. While problematic in some ways, of course, the use of the free version of ArcGIS Online does align with my personal values to allow anyone with broadband connection to use and modify the GIS. To implement all the functions of the GIS on the web, however, required the use a server running Arc Server maintained through the Computer Aided Design and Geographic Information Systems Laboratory (CADGIS) at LSU. An alternative would be a subscription to ArcGIS Online, which allows the same functionality but through cloud computing. LSU, however, does not yet subscribe to ArcGIS Online.

Most of those software packages have extensive help documentation. In cases where that documentation fails to answer questions, a well phrased search of the Web usually yields helpful posts from people who have previously had a similar issue. The following discussions of data sources and methods, therefore, remain relatively general. For step-by-step instructions, see the help documentation for the databases and the specific software packages.

I used a PC and laptop, both with Core i7 processors, to carry out the research.

Data Sources

The data I used are all freely available and, for the most part, broadly accessible.

CLIWOC

The main source of data was the Climatological Database for the World's Oceans, 1750-1850 (CLIWOC), a project funded by the European Union and carried out by a consortium at the Universidad Complutense de Madrid, the University of Sunderland and the University of East Anglia in the UK, the Royal Netherlands Meteorological Institute, and ANIGLA and CONICET in Argentina. The CLIWOC database is freely available for research by anyone and can be downloaded from the project’s website: the “main aim of the project is to produce and make freely available for the scientific community the world’s first daily oceanic climatological database” (http://www.ucm.es/info/cliwoc/intro.htm, accessed December 1, 2012).

The CLIWOC database contains the noon weather observations from 1,674 logbooks that record 4,942 voyages by Spanish, English, French, and Dutch vessels with 975 distinct names, some pertaining to several vessels with the same name, dating to 1750 through 1854. Each weather observation is thereby located by its latitude and longitude on a specific date at noon, allowing climatologists to extend the observational record back in time to well before the establishment of widespread weather stations and climate satellites. CLIWOC 1.0 was released on January 23, 2004 and went through three subsequent versions (1.5, 2.0, and 2.1), each adding records and correcting errors. The last version, CLIWOC 2.1, released on September 25, 2007 and the project has now effectively ended except for continued dissemination of the database. The CLIWOC website also contains an extensive list of references that describe the project, some of them referenced below.

I downloaded CLIWOC 2.1 as a Microsoft Access 2003 database file from the CLIWOC website on November 3, 2011. CLIWOC21_2002-3.mdb contains thirteen tables, with the one named CLIWOC21 containing the bulk of the relevant data in 287,114 rows. The 142 column headings, or fields, are abbreviated but understandable with reference to the detailed explanatory materials on the website and include many variables besides the noontime weather observations and the latitude, longitude: for example, vessel name, type, and nationality; names of the logbook keepers such as the captain and other officers; the prime meridian used; cargoes; landmarks; and origin and destination. The only field not decipherable from the materials on the website related to abbreviations for ownership of the Dutch vessels, some of which (such as WIC for the West India Company) were familiar to me while others were not. An e-mail to Frits Koek at the Royal Netherlands Meteorological Institute, however, rapidly came back with the missing information.

Before using the data in a GIS that suited the particular purposes of my project, I had to delete, combine, and modify many of those fields as well as add some new ones and delete some records. Some of the fields were satisfactory as they stood: for example, the latitudes and longitudes are given as +/- decimal degrees normalized to the present-day prime meridian (the Greenwich meridian) from the hundreds of prime meridians in use before the late nineteenth century. Others required modification or deletion, such as the elimination of the records for many voyages that did not contain any cargo information, achieved by opening the file in MS Access, using the filtering and search functions to create a new table that contained only vessels relevant to the project, and saving the result as CLIWOC21.mdb.

After some similar initial modifications I exported the resulting table as a MS Excel workbook and saved it as CLIWOC21.xlsx to continue modifying the database because I am much more familiar and productive with Excel than Access. I used Excel’s find-and-replace, fill, paste-special, and sort tools as well as the concatenation formula to make various modifications. They include renaming of fields to clarify their meanings; elimination of fields superfluous to my project, such as the one for sea surface temperature; translation of Dutch and Spanish into English; addition of a time field in the format yyyymmdd; and so on.

TSTD

The Trans-Atlantic Slave Trade Database (TSTD) also provided some data, especially the URLs that linked to specific records in that database for vessels carrying enslaved Africans. TSTD contains 34,947 voyages of vessels engaged in the Atlantic slave trade between 1514 and 1866. Unlike CLIWOC, users can not only download the database from the project Website, but also use the Website to search the database by vessel name, origin, destination, and many other fields; generate custom tables and graphs; and view summary graphs, tables, maps, and other representations. Also unlike the CLIWOC database, TSTD does not record vessel locations other than when in port.

TSTD is licensed under a GNU General Public License and a Creative Commons Attribution-Non-Commercial 3.0 and is thereby freely available to copy, distribute, transmit, and adapt the work for non-commercial purposes as long as the data source is attributed to the TSTD and the product is itself is open source. The database and its SPSS codebook are also available to download from the TSTD website as a comma-delimited spreadsheet, but on November 30, 2011 it contained fewer records than available through the search engine of the updated, online database.

To determine which of the CLIWOC voyages also appeared in the TSTD, I used its website search function on November 30, 2011 to create a comma-delimited spreadsheet of all vessel names, downloaded it, opened it in Excel, and named it TSTD_and_CLIWOC_matches.xlsx. The spreadsheet had 34,947 rows, each recording a voyage, with some vessels represented several times because they made more than one voyage. Each row names the vessel as well as, ideally, its captain, the year of the voyage, its origin and destination, the number of slaves aboard, the outcome of the voyage, and so on. I then imported the ShipName, VoyageFrom, VoyageTo, and Name1 (which records the captain’s name) fields from the CLIWOC21 table of CLIWOC21_2002-3.mdb into TSTD_and_CLIWOC_matches.xlsx. Excel’s conditional formatting function then identified all the CLIWOC and TSTD records that had matching vessel and captain names. Parsing the years, origins, and destinations of the matches identified identical voyages in CLIWOC and TSTD. Also, I carefuly checked the spellings of vessel names once I noticed that CLIWOC had retained the spellings of the logbooks while TSTD modernized them: for example, Drie Gezusters instead of Drie Gesusters, Enigheid instead of Eenigheijt, and so on. In the end I had a list of forty-four voyages that appeared in both TSTD and CLIWOC. Since some of the CLIWOC records involved did not have cargo data and I had therefore already eliminated them from my original iteration of CLIWOC21.xlsx, I imported the relevant fields for those records from CLIWOC21_2002-3.mdb into CLIWOC21.xlsx. I also added the data on slaves from TSTD to the cargo field of CLIWOC21.xlsx and created a new field named TRTDURL for the URL that linked to the TSTD search result.

Database of Catalan Voyages

This database includes the voyages of several Catalan vessels, 1837-1900, developed for one of my previous projects: Andrew Sluyter, Black Ranching Frontiers: African Cattle Herders of the Atlantic World, 1500-1900 (Yale University Press, 2012). It derives from my research in four archives in and around Barcelona, Spain in 2010: Arxiu Històric Municipal del Masnou, Arxiu Històric Municipal de Sitges, Biblioteca de Catalunya, and Museu Marítim De Barcelona. From logbooks preserved in those archives, I transcribed the daily noontime position, cargo, and some other data related to tonnage, crews, and captains, but generally not the weather observations, for 21 voyages. For this project I eliminated the voyages without cargo data, leaving 14 voyages by 12 different vessels between 1837 and 1900.

These data are as free as those of CLIWOC and TSTD, although not as widely available. Anyone can go to the relevant archives in Spain and ask to see the logbooks to transcribe the data. But they were not widely available in the same sense as the other databases because they previously could not be downloaded from the Web in a digital format.

HURDAT

Because one of the purposes of the project was to demonstrate how Atlantic voyages interacted with their environment, I also used the North Atlantic Hurricane Database, 1851-2011 (HURDAT). The Hurricane Research Division of the Atlantic Oceanographic and Meteorological Laboratory of the National Oceanic and Atmospheric Administration developed this database, which contains hurricane track and intensity data for the North Atlantic from 1851 through 2011. The National Hurricane Center originally developed HURDAT for 1886-1983, as described in Brian R. Jarvinen, Charles J. Neumann, and Mary A. S. Davis, “A Tropical Cyclone Data Tape for the North Atlantic Basin, 1886-1983: Contents, Limitations, and Uses,” NOAA Technical Memorandum NWS NHC 22 (Miami: National Hurricane Center, 1984). The Atlantic Hurricane Database Re-Analysis Project has since revised HURDAT to extend it back to 1851, continue it forward to the present, and revise all tracks and intensities using additional historical meteorological data, updated models of hurricane behavior, and more sophisticated computing techniques, as described in publications such as C. W. Landsea, C. Anderson, N. Charles, G. Clark, J. Dunion, J. Fernandez-Partagas, P. Hungerford, C. Neumann, and M. Zimmer, “The Atlantic Hurricane Database Re-Analysis Project: Documentation for the 1851-1910 Alterations and Additions to the HURDAT Database,” in Hurricanes and Typhoons: Past, Present and Future, R. J. Murname and K.-B. Liu, eds. (New York City: Columbia University Press, 2004), 177-221.

HURDAT is freely available for non-commercial use, with the stipulation that derivative products must acknowledge the National Oceanic and Atmospheric Administration as the data source: “We ask that a proper acknowledgement to the ‘NOAA Hurricane Research Division of AOML’ accompany the use of these data in any publications or presentations” (http://www.aoml.noaa.gov/hrd/data_sub/datapolicy.html, accessed December 1, 2012).

HURDAT has already been published on ArcGIS Online by NOAA, both as a map service and as a Web map. Details are at http://www.arcgis.com/home/item.html?id=bec3dbc25db14a848427b7f14800395b for the Web map, and at http://www.arcgis.com/home/item.html?id=d9b8b9a1b09b4038813954a1be7043cc for the web service, which is served from the following URL: http://maps4.arcgisonline.com/ArcGIS/rest/services/A-16/NOAA_Hurricane_Tracks-Temporal/MapServer. Anyone can therefore add this layer to their Web map using the free ArcGIS Online viewer and display the data through time using the temporal function. Nonetheless, that choice would not allow control of which specific hurricanes to use in my project, nor how to represent them. Therefore, I chose to download and modify the HURDAT database myself so that I could add hurricanes that I chose and represent them in ways that supported the goals of the project.

On November 30, 2012 I downloaded the file named easytoread-spreadsheet2012-may.xls through the link Excel spreadsheet derived from HURDAT. I used Excel to modify the downloaded file and save it as HURDAT.xlsx for this project. The spreadsheet represents each storm as a series of rows, with each recording the storm position in decimal degrees latitude, longitude as well as its intensity on a particular day at a particular time. The field names are as follows: Month, Day, Hour, Latitude, Longitude, Direction, Speed, Wind, Pressure, and Type. Time is expressed in hours UTC (Coordinated Universal Time), equivalent to GMT (Greenwich Mean Time), meaning the time at the prime meridian.  The time interval between positions is 6 hours: O, 6, 12, and 18 UTC. Positions are given as latitude and longitude expressed in decimal degrees North (N) and West (W). Direction is given as an azimuth, such as 270 degrees. Storm speed and wind speed are both given in miles per hour and kilometers per hour, in separate columns. Pressure is given in millibars but extremely infrequently before the mid twentieth century. Type refers to the intensity category on the widely used Saffir-Simpson Scale.

To prepare the database for the GIS, I used Excel’s find-and-replace, paste-special, and sort tools to make various modifications. I first deleted all records after 1900 so that the worksheet ran from Storm 1 of 1851 through Storm 7 of 1900 (alphabetical naming of storms did not begin until 1950). Second, I added two fields: the first, Name, allowed me to add the storm name to each row, for example, 5 of 1857; the second, Year, added the year of the storm to each row. I also deleted the pressure field; converted the N and W designations of the latitudes and longitudes to + and - preceding the decimal degrees; and converted the month names to numbers, 1 through 12.

Other Databases

Other potential databases were surveyed and explored but not used for various reasons.

For example, The International Comprehensive Ocean-Atmosphere Data Set (ICOADS) is a historical climate database of global marine surface climate observations spanning 1662 to 2008, but its available outputs are gridded at a resolution of 2 degrees latitude by 2 degrees longitude, appropriate as input for historical climate models but not for this project.

Likewise, an inventory of Catalan logbooks by historical climatologists provides data on general itinerary and cargo but not daily position data. The Grup de Climatologia and Grup d’Anàlisi de Situacions Meteorològiques Adverses, Universitat de Barcelona has inventoried 579 logbooks archived in seventeen maritime and other museums in and near Barcelona and begun to assess their potential for reconstructing wind direction and sea condition for the nineteenth-century Atlantic. One of the group, Mariano Barriendos Vallvé, kindly provided me with a copy of that database as a MS Access file (Diarisnavegacio.mdb). That database made it possible to locate relevant logbooks in archives and go to Spain to transcribe them for my database of Catalan voyages.

The Hurricane Research Division of the Atlantic Oceanographic and Meteorological Laboratory of the National Oceanic and Atmospheric Administration also makes available a database similar to HURDAT but for the Eastern and Central North Pacific. I did not use it for this project because it begins only in the mid twentieth century. Its development is described in Mary A. S. Davis, Gail M. Brown, and Preston Leftwich “A Tropical Cyclone Data Tape for the Eastern and Central North Pacific Basins, 1949-1983: Contents, Limitations, and Uses,” NOAA Technical Memorandum NWS NHC 25 (Miami: National Hurricane Center, 1984).

Selected References

1. Elsner, James B., and A. Birol Kara, Hurricanes of the North Atlantic: Climate and Society (Oxford: Oxford University Press, 1999).
2. Eltis, David, and David Richardson, eds., Extending the Frontiers: Essays on the New Transatlantic Slave Trade Database (New Haven: Yale University Press, 2008). 
3. García-Herrera, R., G. P. Können, D. Wheeler, M. R. Prieto, P. D. Jones, and F. B. Koek, CLIWOC: A Climatological Database for the World's Oceans 1750-1854, Climatic Change 73 (2005), 1-12.
4. García-Herrera, R., G. P. Können, D. Wheeler, M. R. Prieto, P. D. Jones, and F. B. Koek, Ship Logbooks Help Analyze Pre-instrumental Climate, EOS 87, no. 18 (2006): 173-180.
5. Können, G. P., and F. B. Koek,. Description of the CLIWOC Database, Climatic Change 73 (2005): 117-130.
6. Prohom Durán, Marc J., and Mariano Barriendos Vallvé, Los diarios de navegación Catalanes: una nueve fuente de datos climáticos sobre los océanos (siglos XVIII a XX), in El Clima Entre el Mar y la Montaña, Juan Carlos García Codrón, Concha Diego Liaño, Pablo Fernández de Arróyabe Hernáez, Carolina Garmendia Pedraja, and Domingo Fernando Rasilla Alvarez, eds., (Santander: Universidad de Cantabria, 2004), pp. 519-28.
7. Prohom Durán, Marc J., El uso de los diarios de navegación como instrumento de reconstrucción climática: la marina catalana del siglo XIX, Investigaciones Geográficas 28 (2002): 89-104.