QKSite Forums

[QKDev]

I have had a look at what is out there that might do the job of collecting site logs and found a copy of a program called Web Log DB that looks promising. It essentially downloads your logs and places them in a database selected by you through ODBC. It also has a scheduler which will run the program at selected intervals to enable you to keep the database up to date. It does cost $69.90 though and doesn't quite do everything I want so I am resigned to writing a program myself to perform the task. I am tempted to make it Open Source and place it up on SourceForge - we'll see...

So here's the outline of what the first bit will be made of...

• Scheduler - To run QKSite or QKSiteLog at selected intervals
  
• FTP Synchroniser - to find the relevant logs and download them
  
• Log Parser - to read the logs and separate out the relevant information. This will work for apache site logs first of all but I am keen to make sure I take care of AdSense and affiliate logs later with this.
  
• Database Storage - decide which kind of database (or leave it to ODBC) to store the logs in.
  
• Log Reporter - I guess I will need to put together a suite of common reports to see what activity has taken place. I want this to be open to SQL or Scripting through QKSite though
  
• Dashboard - Summary information showing trends in your site activity but always available and updated.

More detail on this as I work it through will be placed on the Wiki.

My concerns are currently:

To keep everything as simple and quick as possible
Not to 'over do' it - allow other programs to be used for detailed analysis, so keep it open.
Make a decision on what database to use - SQL Server is preferred, but MySQL may be the better alternative.
Integrate it into QKSite to ensure that any detailed logging can be scripted if we need it.
Finally, to decide on the programming language for all of this. Quite simply my thinking is VB.NET is simple and quick, VC++ is powerful, Java is cross platform and PHP etc is ready for server...

If anyone has any thoughts I would like to hear them. I'll keep this thread up to date with any progress.

Any work on this will be done on the main QKSiteLog page of the wiki.
_________________
QKSite Developer

[QKDev]

Working on this...
I am using the 'monster mash methodology' in searching the open source graveyards for source body parts I can hack together to build the first version - it'll be called the 'It's Alive!!!' version.

The logging applications I am looking through at the moment are detailed at the wiki Window Shopping Open Source Web Site Logging Applications.
_________________
QKSite Developer

[QKDev]

OMG Whatdoyouknow - It died
_________________
QKSite Developer