tsusanto

<POSLog>
   <Transaction>
      <RetailStoreID>48</RetailStoreID>
      <WorkstationID>6</WorkstationID>
      <SequenceNumber>73</SequenceNumber>
      <BusinessDayDate>2014-09-30</BusinessDayDate>
      <EndDateTime>2014-09-30T06:20:14</EndDateTime>
      <OperatorID OperatorName="KERRY P">48237</OperatorID>
      <CurrencyCode>USD</CurrencyCode>
      <RetailTransaction Version="2.1">

tsusanto

<POSLog>
   <Transaction>
      <RetailStoreID>48</RetailStoreID>
      <WorkstationID>6</WorkstationID>
      <SequenceNumber>73</SequenceNumber>
      <BusinessDayDate>2014-09-30</BusinessDayDate>
      <EndDateTime>2014-09-30T06:20:14</EndDateTime>
      <OperatorID OperatorName="KERRY P">48237</OperatorID>
      <CurrencyCode>USD</CurrencyCode>
      <RetailTransaction Version="2.1">

tsusanto

Below is my Flume config file to push files dropped in folder to HDFS
The files are usually about 2MB in size.
The default property deserializer.maxLineLength is set to 2048. Which means after 2048 bytes of data, 
flume truncates the data and treats it as a new event. Thus the resulting file in HDFS had a lot of newlines.
I changed it to 4096000, which is about 4MB



#Flume config file
tier1.sources = xml-source1

tsusanto

//get free account here: https://mongolab.com/plans/
//free plan includes 0.5GB of storage

//refer to http://docs.mongodb.org/manual/applications/crud/

//To list all collections you currently have
db.getCollectionNames();

//To drop a collection
db.testdemo.drop();