Thursday, October 06, 2011

Arduino SD data logger with time stamp

I've been learning about powering my ham shack with solar power and have been working on logging battery voltage to get information about how my power budget is going. At first I used an Arduino's A/D to read the voltage and periodically send it out the USB serial port, but as the shed has no mains power the laptop was a major drain on the battery itself. Yes, measuring was really altering the experiment.

An SD Shield Plus, purchased via eBay, has all the things you need for stand alone data logging. An SD slot, a battery backed clock, and even a bit of non-volatile utility ram.

My Arduino is a slightly older model, the Diecimila and I found that as soon as I called SD.begin() the device would reset. It turns out that the SD library code uses more ram than is available in the older models.

The new Arduino Uno looks like a good improvement. Aside from a more spacious Atmel chip, the USB interface is a processor itself and can be set as a HID device which could be handy. I ordered a couple of Uno's from Little Bird Electronics, and they shipped within minutes, but I still couldn't wait and went to Jaycar to pick up an interesting Uno clone called "Eleven" made by Freetronics.



I hacked together the sample code to write a log file to the SD card with a date time stamp from the clock chip in a format that a spreadsheet will parse. It reads three of the analog inputs and writes a line to the file every ten seconds. Here's the code for future reference:


/*
  SD card datalogger
  by Tom Igoe
  hacked by Peter Marks to write time stamps

 This example code is in the public domain.
 */

#include <SD.h>
// For the SD Shield Plus
const int chipSelect = 10;

// This is to talk to the real time clock
#include "Wire.h"
#define DS1307_I2C_ADDRESS 0x68  // This is the I2C address

// Global Variables
int i;
byte second, minute, hour, dayOfWeek, dayOfMonth, month, year;

void setup()
{
  Wire.begin();
  Serial.begin(9600);
  Serial.print("Initializing SD card...\n");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(chipSelect, OUTPUT);
  Serial.print("chipSelect set to output\n");
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) 
  {
    Serial.println("Card failed, or not present\n");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.\n");
}

void loop()
{
  // make a string for assembling the data to log:
  String dataString = getDateDs1307();
  dataString += String(",");
  
  // read three sensors and append to the string:
  for (int analogPin = 0; analogPin < 3; analogPin++) 
  {
    int sensor = analogRead(analogPin);
    dataString += String(sensor);
    if (analogPin < 2) 
    {
      dataString += ","; 
    }
  }

  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  File dataFile = SD.open("datalog.csv", FILE_WRITE);

  // if the file is available, write to it:
  if (dataFile) 
  {
    dataFile.println(dataString);
    dataFile.close();
    // print to the serial port too:
    Serial.println(dataString);
  }  
  // if the file isn't open, pop up an error:
  else 
  {
    Serial.println("error opening datalog.csv");
  } 
  delay(1000 * 10);
}


// Gets the date and time from the ds1307 and return
// result in a format a spreadsheet can parse: 06/10/11 15:10:00
String getDateDs1307()
{
  // Reset the register pointer
  Wire.beginTransmission(DS1307_I2C_ADDRESS);
  Wire.send(0x00);
  Wire.endTransmission();

  Wire.requestFrom(DS1307_I2C_ADDRESS, 7);

  // A few of these need masks because certain bits are control bits
  second     = bcdToDec(Wire.receive() & 0x7F);
  minute     = bcdToDec(Wire.receive());
  hour       = bcdToDec(Wire.receive() & 0x3F);  
  dayOfWeek  = bcdToDec(Wire.receive());
  dayOfMonth = bcdToDec(Wire.receive());
  month      = bcdToDec(Wire.receive());
  year       = bcdToDec(Wire.receive());
  
  String dataString = "";
  
  dataString += Print2Digit(dayOfMonth); 
  dataString += String("/");
  dataString += Print2Digit(bcdToDec(month)); 
  dataString += String("/"); // Y2k1 bug!
  dataString += Print2Digit(bcdToDec(year));
  dataString += String(" ");
  dataString += Print2Digit(hour);
  dataString += String(":");
  dataString += Print2Digit(minute);
  dataString += String(":");
  dataString += Print2Digit(second);

  return dataString;
}

// Convert binary coded decimal to normal decimal numbers
byte bcdToDec(byte val)
{
  return ( (val/16*10) + (val%16) );
}

String Print2Digit(byte Val)
{
  String dataString = "";
  if (Val < 10)
  {
    dataString = "0";
  }  
  dataString += String(Val, DEC);
  return dataString;
}

The output file looks like this:


06/10/11 15:36:16,555,489,479
06/10/11 15:36:18,461,427,438
06/10/11 15:36:19,419,401,414
06/10/11 15:36:21,397,387,404
06/10/11 15:36:22,394,389,403
06/10/11 15:36:23,380,375,393
06/10/11 15:36:25,386,383,399
06/10/11 15:36:26,382,380,396
06/10/11 15:36:28,380,378,395
06/10/11 15:36:29,387,385,400


Here's a few days of data. Today was partially cloudy but the battery was re-charged in the end.


Plotted with python and matplotlib with this:


#!/usr/local/bin/python

import datetime
import matplotlib
matplotlib.use('Agg')

import matplotlib.pyplot as plt
import csv

INFILENAME = 'datalog.csv'
VOLTSCALE = 665.0 / 13.7

def main():
    datareader = csv.reader(open(INFILENAME,'r'), delimiter=',')
    dates = []
    points = []
    for row in datareader:
        if len(row) > 1:
            #Converts str into a datetime object. 08/10/11 12:23:12
            dates.append(datetime.datetime.strptime(row[0],'%d/%m/%y %H:%M:%S'))
            volts = float(row[1]) / VOLTSCALE
            points.append(volts)
    
    fig = plt.figure()
    plt.plot(dates, points)
    plt.title("Battery Voltage over several days")
    plt.ylabel('Battery Volts')
    fig.autofmt_xdate()
    plt.ylim(ymin=8)
    plt.show()
    plt.savefig('voltgraph')

if __name__ == "__main__":
    main()
    

12 comments:

  1. Hey I use processing with the arduino to graph and save my data. I've made a real time monitoring graph and save also the data on excel in case of further analysis.

    Now this program matplotlib looks really nice. I want to use but it doesnt look like it comes free. im downloading,
    The Enthought Python Distribution
    which is released under a less restrictive license than the (full) EPD. do you know where I can download a full or is this restricted enough for just saving and plotting data.

    Also with processing I am planning on saving data in excel with timestamps in a column and graphing both real time with the timestamp. However I dont have any sd card shield. I googled a bit to find the function for timestamp
    and came across this Date d = new Date();
    long current = d.getTime()/1000;
    do you knw about this?

    My real time montoring is no where near as good as yours. I'd love to try your code for my project.

    ReplyDelete
  2. Matplotlib is free so give it a try. I don't know what Date() does without a clock chip, maybe it starts when you power up.

    Best wishes,

    Peter

    ReplyDelete
  3. I am very sorry to bother you about this, but the links I've googled so far are working properly on my computer.

    Can you please send me a link on where I can download this program for windows, because I've googled a few websites and I cant find any proper link.

    ReplyDelete
  4. You can find a matplotlib installer for windows at: http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.1.1/

    It's a python module so you'll also have to install python.

    I'm sorry I don't use Windows so I can't really be of much help there. Consider getting an old PC and installing linux on it - everything works much better in my experience.

    Peter

    ReplyDelete
  5. Hazim Jauhari5:09 PM

    hi.. may i know is it this arduino code used an additional library? because when i want to compile it, it shows error...

    ReplyDelete
  6. What error do you get?

    ReplyDelete
  7. Hello Peter,

    thanks for posting your code, I'm also getting an error, Arduino Compiler is expecting a file after your first #include, please advise on the filename.

    Regards,

    Juan

    ReplyDelete
  8. Sorry about that Juane and Hazim.

    I've updated the code to include SD.h which comes on the CD with the SD Shield plus.

    Peter

    ReplyDelete
  9. Hi Mark

    I have been looking at your code and changed it to suit a project I'm working on. I was wondering what your thoughts were on the Data Logger Shield? Did it take long to arrive from Thailand?

    Cheers

    Jase

    ReplyDelete
  10. Hi Tepper,

    It's a good shield. I don't really remember how long it took to arrive, so I guess it was reasonable.

    Peter

    ReplyDelete
  11. date & time issue how can I solve it ?

    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,30,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0
    165/105/105 45:165:85,0,0,0

    ReplyDelete
  12. Hey Peter,

    Great code, the only issue I'm having is that I'm getting an error saying that the "Wire" library doesn't have the members 'send'and 'receive'? Any ideas on how to get around this?

    Thanks in advance.

    ReplyDelete