mainSavannah Administration - Support: sr #106700, Page statistics/log files?...

 
 

sr #106700: Page statistics/log files? revisited

Submitter:  Alex Fernandez <alexfernandez>
Submitted:  Sun 29 Mar 2009 05:05:11 PM UTC
   
 
Category:  Project webpages Priority:  5 - Normal
Severity:  4 - Important Status:  None
Privacy:  Public Assigned to:  None
Operating System:  GNU/Linux Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

  Spam posted by anonymous
Sun 12 Apr 2009 09:28:54 PM UTC, comment #12: 

Hi Alex -- I don't want to delete the information from the file, since it's all we have to go on for future research.  I added notes about the non-mirroring and bad address.  Thanks.

Karl Berry <karl>
Site Administrator
Sun 12 Apr 2009 05:39:06 PM UTC, comment #11: 

A couple of addresses from /root/mirrors-contacts.txt are outdated:


I don't have root permissions to update the info. Thanks!

Alex Fernandez <alexfernandez>
Sun 12 Apr 2009 05:04:13 PM UTC, comment #10: 

Hi,

You are right, disabling DNS lookups meant reducing the 82 minutes to little more than five.

I think that the reports that mainly interest developers are the urldetail ones, to see how many accesses each version and page have had. (At least it is so for me.) Therefore, awstats would be adequate.

I am going to contact -email is unavailable- and cc you. For mirror maintainers I will write them and see what we can do.

Alex Fernandez <alexfernandez>
Mon 06 Apr 2009 08:52:57 PM UTC, comment #9: 

Speaking as one of the gnu.org webmasters, I can tell you we have never made any attempt to get logs (or anything else) from the mirror maintainers.  We're just happy when they work :).


Karl Berry <karl>
Site Administrator
Mon 06 Apr 2009 08:43:40 PM UTC, comment #8: 

Hi,

For awstats I think you need to disable the DNS lookups (check /root/infra/download.txt for more information about the current awstats setup, btw), it will quicken the processing.

A problem with this method is that not all reports can be generated (e.g. only those that support the :filter syntax).

We don't have a special setup with mirror maintainers: they synchronize the download area using rsync access, which is available to everybody. How to get their logs is something to decide. You might want to ask -email is unavailable- if by any chance they did something for ftp.gnu.org mirrors.

For -email is unavailable- this should be easier because they have access to Savannah and could rsync files nightly (for example).

Sylvain Beucler <Beuc>
Fri 03 Apr 2009 09:42:22 PM UTC, comment #7: 

Today I did the test with awstats. Processing the huge Apache2 log from download (sftp), with 1385993 records, took 82m33.840s. (About 300 records per second, which seems pretty slow.) However once it is done you can extract all the relevant pages very fast. After that I run the test:

./awstats.pl -config=elyxer -staticlinks -output=urldetail:/releases-noredirect/elyxer

The results were quite good too. In fact a global config file can be used, and then individual reports generated automatically for every project.

A sample result page is at ~/awstats-test/elyxer-stats-200903.html

Therefore both products can be made to work in our particular situation. I would recommend awstats over webalizer because it is easier to aggregate results from different providers (at least if everyone is using the same log format, which is not a given.) With webalizer these results have to be sorted first and then aggregated.

I am ready now to contact all mirror maintainers and sysadmin@gnu.org, and ask them to place their log files in some available directory. How are we supposed to get them securely? Do they have our public key, or something?

Let me know how we are going to proceed.

Alex Fernandez <alexfernandez>
Mon 30 Mar 2009 11:10:18 PM UTC, comment #6: 

Ok, I spent some time browsing the docs. Webalizer per se cannot split access files to different outputs, but there are at least three different solutions (probably more):

  • Use webalizer on a subset of the logs. A simple grep can filter the lines (e.g. with "GET /releases-noredirect/elyxer/") and the result can be fed to webalizer.


cat access.log | grep "GET /releases-noredirect/elyxer/" > elyxer-access.log
webalizer elyxer-access.log

  • Use awstats with an appropriate filter.


perl awstats.pl -config=elyxer -output=urldetail:/releases-noredirect/elyxer -staticlinks > awstats.elyxer.urldetailwithfilter.html

Should be easy enough to do and I believe could be automated.

  • A homegrown solution. Nothing fancy, just a bash script that counts the number of times each file appears and outputs it with some format. Probably the worst of the three as it may need more maintenance.


I have run a silly test of the first option (webalizer) and it is pretty fast: even with the current access.log for download.sftp it took a couple of seconds, the results are at webalizer-test/index.html in svadmin's home.

The second option (awstats) probably would require more effort but the results might be quite better. When I have some more time I will try another silly test.

Alex Fernandez <alexfernandez>
Mon 30 Mar 2009 06:41:39 PM UTC, comment #5: 

That sounds good.

Getting stats from the mirrors is essentially asking the mirror maintainers (/root/mirrors-contacts.txt) to make them available somewhere for us to grab. I believe sourceforge does the same for its downloads stats.

Can webalizer automatically split stats for each project?

Sylvain Beucler <Beuc>
Sun 29 Mar 2009 11:13:37 PM UTC, comment #4: 

Great! That is a quite nifty configuration you have there.

Anyway, I have taken a look and I think I have some grasp on the problem right now. It can be divided along the lines you suggested below.

The first part (download statistics) is not as easy as I would have liked but it should be workable: make webalizer aware of the many projects in the sftp vserver, so that it tracks downloads for every project (without bogging down the whole machine).

The second part (website statistics) requires cooperation of -email is unavailable- to get hold of their logs. After that some project separation (similar to the one in download) will probably be required.

The third part (mirror download statistics) is even more complex and will probably require some setup on their part.

Does it look ok to you? Will keep you posted of any progress.

Alex Fernandez <alexfernandez>
Sun 29 Mar 2009 07:29:13 PM UTC, comment #3: 

For a start we'll grant you access to svadmin@sv.gnu.org, a "read-only" account from where you'll be able to study the current configuration.

Please update the comments .ssh/authorized_keys with your contact information.

Then you'll want to read http://savannah.gnu.org/maintenance/SavannahArchitecture and /root/infra/vservers.txt

You can also join #savannah, check http://savannah.gnu.org/maintenance/SavannahHackersCommunication

I propose you study the Savannah setup and then we'll discuss stats some more :)

Sylvain Beucler <Beuc>
Sun 29 Mar 2009 06:50:34 PM UTC, comment #2: 

I think that is great. For the record I am a fellow of FSFE, the details are in a private mail, but we can wait for a month, no problem.

I was planning on using webalizer, given the awful security track record of awstats. I think it can be configured to give stats for different subsites -- in fact it is what gna.org are using: http://stats.gna.org/homepage.gna.org/admin/usage_200902.html. I don't think that -email is unavailable- would have any trouble with it, but I will ask first. Since you already have awstats it might also be an option.

Stats should be accesible on a per-project basis at http://www.nongnu.org/elyxer/stats.html or something similar.

As to mirror owners I think it is a hard task to coalesce logs from all mirrors into a single page. I would start with stats for no-redirect and follow from there.

Alex Fernandez <alexfernandez>
Sun 29 Mar 2009 06:06:09 PM UTC, comment #1: 

Hi,

Your help offer is greatly appreciated.
How do you plan to add this feature?

Currently we can't give you access to Savannah because we don't know you yet, but if you show long-term (e.g. 1 month) interest and support this can be arranged.

For this feature I see 2 different tasks that needs to be implemented:

- technical: split the logs efficiently while keeping the main log as-is (so we can keep the site-wide stats like http://download.savannah.gnu.org/awstats.pl)

- contact 3 groups of people involved in the logs


How about it?

Sylvain Beucler <Beuc>
Sun 29 Mar 2009 05:05:11 PM UTC, original submission:  

I would like to see page statistics for the hosted project elyxer, especially for the project website and for downloads. I am thinking of something along the lines of awstats or webalizer.

For quite a few years there has been a support request pending for site statistics: [https://savannah.gnu.org/task/index.php?3492]. But I don't know if there is something already implemented; I could not find it. I have some experience in Debian administration and if necessary would be willing to volunteer some time for this task so that all projects can see their web/ftp statistics.

Alex Fernandez <alexfernandez>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by karl (Posted a comment)
  • -email is unavailable- added by Beuc (Posted a comment)
  • -email is unavailable- added by alexfernandez (Submitted the item)
  • -email is unavailable- added by alexfernandez
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-03-29 alexfernandez Carbon-Copy- Added alexfernandez

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code