Skip to content
Extraits de code Groupes Projets
Valider 4ecaa3f0 rédigé par Benaka's avatar Benaka
Parcourir les fichiers

Merge pull request #7461 from piwik/7163_log_analytics_repo 

Fixes #7163, removing misc/log-analytics directory and replace w/ submodule.
parents 74f0d0ff 5344fec1
Aucune branche associée trouvée
Aucune étiquette associée trouvée
Aucune requête de fusion associée trouvée
Tout étendre Masquer les modifications d'espaces
En ligne Côte à côte
Affichage de
avec 4 ajouts et 3130 suppressions
.gitmodules
+ 3
− 0
Voir le fichier @ 4ecaa3f0
...@@ -39,3 +39,6 @@ ...@@ -39,3 +39,6 @@
path = libs/PiwikTracker path = libs/PiwikTracker
url = https://github.com/piwik/piwik-php-tracker.git url = https://github.com/piwik/piwik-php-tracker.git
branch = master branch = master
[submodule "misc/log-analytics"]
path = misc/log-analytics
url = https://github.com/piwik/piwik-log-analytics.git
log-analytics @ 5afda62f
Subproject commit 5afda62fcd79479e17ea3d81a91030593fbdfc45
misc/log-analytics/README.md supprimé 100644 → 0
+ 0
− 287
Voir le fichier @ 74f0d0ff
# Piwik Server Log Analytics: Import your server logs in Piwik!
## Requirements
* Python 2.6 or 2.7. Python 3.x is not supported.
* Update to Piwik 1.11
## Contributors
We're looking for contributors! Feel free to submit Pull requests on Github.
For example this documentation page could be improved and maybe you would like to help? Or **maybe you know Python**, check out the [list of issues for import_logs.py](https://github.com/piwik/piwik/labels/c%3A%20Log%20Analytics%20%28import_logs.py%29) which lists many interesting ideas and projects that need help. FYI [we plan to move](https://github.com/piwik/piwik/issues/7163) the project to its own repository on Github and split the big file into smaller files.
## How to use this script?
The most simple way to import your logs is to run:
./import_logs.py --url=piwik.example.com /path/to/access.log
You must specify your Piwik URL with the `--url` argument.
The script will automatically read your config.inc.php file to get the authentication
token and communicate with your Piwik install to import the lines.
The default mode will try to mimic the Javascript tracker as much as possible,
and will not track bots, static files, or error requests.
If you wish to track all requests the following command would be used:
python /path/to/piwik/misc/log-analytics/import_logs.py --url=http://mysite/piwik/ --idsite=1234 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots access.log
### Format Specific Details
* If you are importing Netscaler log files, make sure to specify the **--iis-time-taken-secs** option. Netscaler stores
the time-taken field in seconds while most other formats use milliseconds. Using this option will ensure that the
log importer interprets the field correctly.
## How to import your logs automatically every day?
You must first make sure your logs are automatically rotated every day. The most
popular ways to implement this are using either:
* logrotate: http://www.linuxcommand.org/man_pages/logrotate8.html
It will work with any HTTP daemon.
* rotatelogs: http://httpd.apache.org/docs/2.0/programs/rotatelogs.html
Only works with Apache.
* let us know what else is useful and we will add it to the list
Your logs should be automatically rotated and stored on your webserver, for instance in daily logs
`/var/log/apache/access-%Y-%m-%d.log` (where %Y, %m and %d represent the year,
month and day).
You can then import your logs automatically each day (at 0:01). Setup a cron job with the command:
0 1 * * * /path/to/piwik/misc/log-analytics/import-logs.py -u piwik.example.com `date --date=yesterday +/var/log/apache/access-\%Y-\%m-\%d.log`
## Performance
With an Intel Core i5-2400 @ 3.10GHz (2 cores, 4 virtual cores with Hyper-threading),
running Piwik and its MySQL database, between 250 and 300 records were imported per second.
The import_logs.py script needs CPU to read and parse the log files, but it is actually
Piwik server itself (i.e. PHP/MySQL) which will use more CPU during data import.
To improve performance,
1. by default, the script one thread to parse and import log lines.
you can use the `--recorders` option to specify the number of parallel threads which will
import hits into Piwik. We recommend to set `--recorders=N` to the number N of CPU cores
that the server hosting Piwik has. The parsing will still be single-threaded,
but several hits will be tracked in Piwik at the same time.
2. the script will issue hundreds of requests to piwik.php - to improve the Piwik webserver performance
you can disable server access logging for these requests.
Each Piwik webserver (Apache, Nginx, IIS) can also be tweaked a bit to handle more req/sec.
## Advanced uses
### Example Nginx Virtual Host Log Format
This log format can be specified for nginx access logs to capture multiple virtual hosts:
* log_format vhosts '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
* access_log /PATH/TO/access.log vhosts;
When executing import_logs.py specify the "common_complete" format.
### How do I import Page Speed Metric from logs?
In Piwik> Actions> Page URLs and Page Title reports, Piwik reports the Avg. generation time, as an indicator of your website speed.
This metric works by default when using the Javascript tracker, but you can use it with log file as well.
Apache can log the generation time in microseconds using %D in the LogFormat.
This metric can be imported using a custom log format in this script.
In the command line, add the --log-format-regex parameter that contains the group generation_time_micro.
Here's an example:
Apache LogFormat "%h %l %u %t \"%r\" %>s %b %D"
--log-format-regex="(?P<ip>\S+) \S+ \S+ \[(?P<date>.*?) (?P<timezone>.*?)\] \"\S+ (?P<path>.*?) \S+\" (?P<status>\S+) (?P<length>\S+) (?P<generation_time_micro>\S+)"
Note: the group <generation_time_milli> is also available if your server logs generation time in milliseconds rather than microseconds.
### How do I setup Nginx to directly imports in Piwik via syslog?
With the syslog patch from http://wiki.nginx.org/3rdPartyModules which is compiled in dotdeb's release, you can log to syslog and imports them live to Piwik.
Path: Nginx -> syslog -> (syslog central server) -> this script -> piwik
You can use any log format that this script can handle, like Apache Combined, and Json format which needs less processing.
##### Setup Nginx logs
```
http {
...
log_format piwik '{"ip": "$remote_addr",'
'"host": "$host",'
'"path": "$request_uri",'
'"status": "$status",'
'"referrer": "$http_referer",'
'"user_agent": "$http_user_agent",'
'"length": $bytes_sent,'
'"generation_time_milli": $request_time,'
'"date": "$time_iso8601"}';
...
server {
...
access_log syslog:info piwik;
...
}
}
```
##### Setup syslog-ng
This is the config for the central server if any. If not, you can also use this config on the same server as Nginx.
```
options {
stats_freq(600); stats_level(1);
log_fifo_size(1280000);
log_msg_size(8192);
};
source s_nginx { udp(); };
destination d_piwik {
program("/usr/local/piwik/piwik.sh" template("$MSG\n"));
};
log { source(s_nginx); filter(f_info); destination(d_piwik); };
```
##### piwik.sh
Just needed to configure the best params for import_logs.py :
```
#!/bin/sh
exec python /path/to/misc/log-analytics/import_logs.py \
--url=http://localhost/ --token-auth=<your_auth_token> \
--idsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots \
--log-format-name=nginx_json -
```
##### Example of regex for syslog format (centralized logs)
###### log format exemple
```
Aug 31 23:59:59 tt-srv-name www.tt.com: 1.1.1.1 - - [31/Aug/2014:23:59:59 +0200] "GET /index.php HTTP/1.0" 200 3838 "http://www.tt.com/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0" 365020 www.tt.com
```
###### Corresponding regex
```
--log-format-regex='.* ((?P<ip>\S+) \S+ \S+ \[(?P<date>.*?) (?P<timezone>.*?)\] "\S+ (?P<path>.*?) \S+" (?P<status>\S+) (?P<length>\S+) "(?P<referrer>.*?)" "(?P<user_agent>.*?)").*'
```
### Setup Apache CustomLog that directly imports in Piwik
Since apache CustomLog directives can send log data to a script, it is possible to import hits into piwik server-side in real-time rather than processing a logfile each day.
This approach has many advantages, including real-time data being available on your piwik site, using real logs files instead of relying on client-side Javacsript, and not having a surge of CPU/RAM usage during log processing.
The disadvantage is that if Piwik is unavailable, logging data will be lost. Therefore we recommend to also log into a standard log file. Bear in mind also that apache processes will wait until a request is logged before processing a new request, so if piwik runs slow so does your site: it's therefore important to tune --recorders to the right level.
##### Basic setup
You might have in your main config section:
```
# Set up your log format as a normal extended format, with hostname at the start
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" myLogFormat
# Log to a file as usual
CustomLog /path/to/logfile myLogFormat
# Log to piwik as well
CustomLog "|/path/to/import_logs.py --option1 --option2 ... -" myLogFormat
```
Note: on Debian/Ubuntu, the default configuration defines the vhost_combined format. You
can use it instead of defining myLogFormat.
Useful options here are:
* --add-sites-new-hosts (creates new websites in piwik based on %v in the LogFormat)
* --output=/path/to/piwik.log (puts any output into a log file for reference/debugging later)
* --recorders=4 (use whatever value seems sensible for you - higher traffic sites will need more recorders to keep up)
* "-" so it reads straight from /dev/stdin
You can have as many CustomLog statements as you like. However, if you define any CustomLog directives within a <VirtualHost> block, all CustomLogs in the main config will be overridden. Therefore if you require custom logging for particular VirtualHosts, it is recommended to use mod_macro to make configuration more maintainable.
##### Advanced setup: Apache vhost, custom logs, automatic website creation
As a rather extreme example of what you can do, here is an apache config with:
* standard logging in the main config area for the majority of VirtualHosts
* customised logging in a particular virtualhost to change the hostname (for instance, if a particular virtualhost should be logged as if it were a different site)
* customised logging in another virtualhost which creates new websites in piwik for subsites (e.g. to have domain.com/subsite1 as a whole website in its own right). This requires setting up a custom --log-format-regex to allow "/" in the hostname section (NB the escaping necessary for apache to pass through the regex to piwik properly), and also to have multiple CustomLog directives so the subsite gets logged to both domain.com and domain.com/subsite1 websites in piwik
* we also use mod_rewrite to set environment variables so that if you have multiple subsites with the same format , e.g. /subsite1, /subsite2, etc, you can automatically create a new piwik website for each one without having to configure them manually
NB use of mod_macro to ensure consistency and maintainability
Apache configuration source code:
```
# Set up macro with the options
# * $vhost (this will be used as the piwik website name),
# * $logname (the name of the LogFormat we're using),
# * $output (which logfile to save import_logs.py output to),
# * $env (CustomLog can be set only to fire if an environment variable is set - this contains that environment variable, so subsites only log when it's set)
# NB the --log-format-regex line is exactly the same regex as import_logs.py's own 'common_vhost' format, but with "\/" added in the "host" section's allowed characters
<Macro piwiklog $vhost $logname $output $env>
LogFormat "$vhost %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" $logname
CustomLog "|/path/to/piwik/misc/log-analytics/import_logs.py \
--add-sites-new-hosts \
--config=/path/to/piwik/config/config.ini.php \
--url='http://your.piwik.install/' \
--recorders=4 \
--log-format-regex='(?P<host>[\\\\w\\\\-\\\\.\\\\/]*)(?::\\\\d+)? (?P<ip>\\\\S+) \\\\S+ \\\\S+ \\\\[(?P<date>.*?) (?P<timezone>.*?)\\\\] \\\"\\\\S+ (?P<path>.*?) \\\\S+\\\" (?P<status>\\\\S+) (?P<length>\\\\S+) \\\"(?P<referrer>.*?)\\\" \\\"(?P<user_agent>.*?)\\\"' \
--output=/var/log/piwik/$output.log \
-" \
$logname \
$env
</Macro>
# Set up main apache logging, with:
# * normal %v as hostname,
# * vhost_common as logformat name,
# * /var/log/piwik/main.log as the logfile,
# * no env variable needed since we always want to trigger
Use piwiklog %v vhost_common main " "
<VirtualHost>
ServerName example.com
# Set this host to log to piwik with a different hostname (and using a different output file, /var/log/piwik/example_com.log)
Use piwiklog "another-host.com" vhost_common example_com " "
</VirtualHost>
<VirtualHost>
ServerName domain.com
# We want to log this normally, so repeat the CustomLog from the main section
# (if this is omitted, our other CustomLogs below will override the one in the main section, so the main site won't be logged)
Use piwiklog %v vhost_common main " "
# Now set up mod_rewrite to detect our subsites and set up new piwik websites to track just hits to these (this is a bit like profiles in Google Analytics).
# We want to match domain.com/anothersubsite and domain.com/subsite[0-9]+
# First to be on the safe side, unset the env we'll use to test if we're in a subsite:
UnsetEnv vhostLogName
# Subsite definitions. NB check for both URI and REFERER (some files used in a page, or downloads linked from a page, may not reside within our subsite directory):
# Do the one-off subsite first:
RewriteCond %{REQUEST_URI} ^/anothersubsite(/|$) [OR]
RewriteCond %{HTTP_REFERER} domain\.com/anothersubsite(/|$)
RewriteRule ^/.* - [E=vhostLogName:anothersubsite]
# Subsite of the form /subsite[0-9]+. NB the capture brackets in the RewriteCond rules which get mapped to %1 in the RewriteRule
RewriteCond %{REQUEST_URI} ^/(subsite[0-9]+)(/|$)) [OR]
RewriteCond %{HTTP_REFERER} domain\.com/(subsite[0-9]+)(/|$)
RewriteRule ^/.* - [E=vhostLogName:subsite%1]
# Now set the logging to piwik setting:
# * the hostname to domain.com/<subsitename>
# * the logformat to vhost_domain_com_subsites (can be anything so long as it's unique)
# * the output to go to /var/log/piwik/domain_com_subsites.log (again, can be anything)
# * triggering only when the env variable is set, so requests to other URIs on this domain don't call this logging rule
Use piwiklog domain.com/%{vhostLogName}e vhost_domain_com_subsites domain_com_subsites env=vhostLogName
</VirtualHost>
```
### And that's all !
***This documentation is a community effort, feel free to suggest any change via Github Pull request.***
misc/log-analytics/import_logs.py supprimé 100755 → 0
+ 0
− 2104
Voir le fichier @ 74f0d0ff
Ce diff est replié.
misc/log-analytics/tests/logs/amazon_cloudfront_rtmp.log supprimé 100644 → 0
+ 0
− 4
Voir le fichier @ 74f0d0ff
#Version: 1.0
#Fields: date time x-edge-location c-ip x-event sc-bytes x-cf-status x-cf-client-id cs-uri-stem cs-uri-query c-referrer x-page-url​ c-user-agent x-sname x-sname-query x-file-ext x-sid
2010-03-12 23:51:20 SEA4 192.0.2.147 connect 2014 OK bfd8a98bee0840d9b871b7f6ade9908f rtmp://shqshne4jdp4b6.cloudfront.net/cfx/st​ key=value http://player.longtailvideo.com/player.swf http://www.longtailvideo.com/support/jw-player-setup-wizard?example=204 LNX%2010,0,32,18 - - - -
2010-03-12 23:51:21 SEA4 192.0.2.222 play 3914 OK bfd8a98bee0840d9b871b7f6ade9908f rtmp://shqshne4jdp4b6.cloudfront.net/cfx/st​ key=value http://player.longtailvideo.com/player.swf http://www.longtailvideo.com/support/jw-player-setup-wizard?example=204 LNX%2010,0,32,18 myvideo p=2&q=4 flv 1
misc/log-analytics/tests/logs/amazon_cloudfront_web.log supprimé 100644 → 0
+ 0
− 3
Voir le fichier @ 74f0d0ff
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken
2014-05-23 01:13:11 FRA2 182 192.0.2.10 GET d111111abcdef8.cloudfront.net /view/my/file.html 200 www.displaymyfiles.com Mozilla/4.0%20(compatible;%20MSIE%205.0b1;%20Mac_PowerPC) - zip=98101 RefreshHit MRVMF7KydIvxMWfJIglgwHQwZsbG2IhRJ07sn9AkKUFSHS9EXAMPLE== d111111abcdef8.cloudfront.net http - 0.001
misc/log-analytics/tests/logs/common.log supprimé 100644 → 0
+ 0
− 1
Voir le fichier @ 74f0d0ff
1.2.3.4 - - [10/Feb/2012:16:42:07 -0500] "GET / HTTP/1.0" 301 368
misc/log-analytics/tests/logs/common_complete.log supprimé 100644 → 0
+ 0
− 1
Voir le fichier @ 74f0d0ff
www.example.com 1.2.3.4 - - [10/Feb/2012:16:42:07 -0500] "GET / HTTP/1.0" 301 368 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
misc/log-analytics/tests/logs/common_vhost.log supprimé 100644 → 0
+ 0
− 1
Voir le fichier @ 74f0d0ff
www.example.com 1.2.3.4 - - [10/Feb/2012:16:42:07 -0500] "GET / HTTP/1.0" 301 368
misc/log-analytics/tests/logs/icecast2.log supprimé 100644 → 0
+ 0
− 1
Voir le fichier @ 74f0d0ff
1.2.3.4 - - [10/Feb/2012:16:42:07 -0500] "GET / HTTP/1.0" 301 368 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11" 1807
\ No newline at end of file
misc/log-analytics/tests/logs/iis.log supprimé 100644 → 0
+ 0
− 5
Voir le fichier @ 74f0d0ff
#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2012-04-01 00:00:13
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2012-04-01 00:00:13 W3SVC834221556 PXQD1 1.2.3.4 GET /foo/bar topCat1=divinity&submit=Search 80 theuser 5.6.7.8 HTTP/1.1 Mozilla/5.0+(X11;+U;+Linux+i686;+en-US;+rv:1.9.2.7)+Gecko/20100722+Firefox/3.6.7 - - example.com 200 654 456 27028 214 1687
misc/log-analytics/tests/logs/iis_custom.log supprimé 100644 → 0
+ 0
− 7
Voir le fichier @ 74f0d0ff
#Software: IIS Advanced Logging Module
#Version: 1.0
#Start-Date: 2014-11-18 00:00:00.128
#Fields: date-local time-local s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) cs(Host) sc-status sc-substatus sc-win32-status TimeTakenMS
2012-08-15 17:00:00.363 10.10.28.140 GET /Products/theProduct - 80 - "70.95.0.0" "Mozilla/5.0 (Linux; Android 4.4.4; SM-G900V Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.59 Mobile Safari/537.36" "http://example.com/Search/SearchResults.pg?informationRecipient.languageCode.c=en" "xzy.example.com" 200 0 0 109
2012-08-15 17:00:00.660 10.10.28.140 GET /Topic/hw43061 - 80 - "70.95.32.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36" - "example.hello.com" 301 0 0 0
2012-08-15 17:00:00.675 10.10.28.140 GET /hello/world/6,681965 - 80 - "173.5.0.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36" - "hello.example.com" 404 0 0 359
misc/log-analytics/tests/logs/logs_to_tests.log supprimé 100644 → 0
+ 0
− 6
Voir le fichier @ 74f0d0ff
203.38.78.246 - - [05/Feb/2013:07:01:26 +0000] "GET /piwik.php?action_name=Clearcode%20-%20Web%20and%20Mobile%20Development%20%7C%20Technology%20With%20Passion&idsite=1&rec=1&r=983420&h=17&m=31&s=25&url=http%3A%2F%2Fclearcode.cc%2F&urlref=http%3A%2F%2Fclearcode.cc%2Fwelcome&_id=1da79fc743e8bcc4&_idts=1360047661&_idvc=1&_idn=0&_refts=1360047661&_viewts=1360047661&_ref=http%3A%2F%2Fpiwik.org%2Fthank-you-all%2F&pdf=1&qt=1&realp=0&wma=1&dir=1&fla=1&java=1&gears=0&ag=1&cookie=1&res=1680x1050 HTTP/1.1" 200 192 "http://clearcode.cc/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17"
203.38.78.246 - - [05/Feb/2013:07:01:41 +0000] "GET /piwik.php?action_name=AdviserBrief%20-%20Track%20Your%20Investments%20and%20Plan%20Financial%20Future%20%7C%20Clearcode&idsite=1&rec=1&r=109464&h=17&m=31&s=40&url=http%3A%2F%2Fclearcode.cc%2Fcase%2Fadviserbrief-track-your-investments-and-plan-financial-future%2F&urlref=http%3A%2F%2Fclearcode.cc%2Fwelcome&_id=1da79fc743e8bcc4&_idts=1360047661&_idvc=1&_idn=0&_refts=1360047661&_viewts=1360047661&_ref=http%3A%2F%2Fpiwik.org%2Fthank-you-all%2F&pdf=1&qt=1&realp=0&wma=1&dir=1&fla=1&java=1&gears=0&ag=1&cookie=1&res=1680x1050 HTTP/1.1" 200 192 "http://clearcode.cc/case/adviserbrief-track-your-investments-and-plan-financial-future" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17"
203.38.78.246 - - [05/Feb/2013:07:01:46 +0000] "GET /piwik.php?action_name=ATL%20Apps%20-%20American%20Tailgating%20League%20Mobile%20Android%20IOS%20Games%20%7C%20Clearcode&idsite=1&rec=1&r=080064&h=17&m=31&s=46&url=http%3A%2F%2Fclearcode.cc%2Fcase%2Fatl-apps-mobile-android-ios-games%2F&urlref=http%3A%2F%2Fclearcode.cc%2Fwelcome&_id=1da79fc743e8bcc4&_idts=1360047661&_idvc=1&_idn=0&_refts=1360047661&_viewts=1360047661&_ref=http%3A%2F%2Fpiwik.org%2Fthank-you-all%2F&pdf=1&qt=1&realp=0&wma=1&dir=1&fla=1&java=1&gears=0&ag=1&cookie=1&res=1680x1050 HTTP/1.1" 200 192 "http://clearcode.cc/case/atl-apps-mobile-android-ios-games" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17"
# Request with invalid length should not error out
95.81.66.139 - - [29/Oct/2013:00:48:46 +0100] "HEAD / HTTP/1.0" 403 - "-" "-"
\ No newline at end of file
misc/log-analytics/tests/logs/ncsa_extended.log supprimé 100644 → 0
+ 0
− 1
Voir le fichier @ 74f0d0ff
1.2.3.4 - - [10/Feb/2012:16:42:07 -0500] "GET / HTTP/1.0" 301 368 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
misc/log-analytics/tests/logs/netscaler.log supprimé 100644 → 0
+ 0
− 5
Voir le fichier @ 74f0d0ff
#Version: 1.0
#Software: Netscaler Web Logging(NSWL)
#Date: 2014-02-18 11:55:13
#Fields: date time c-ip cs-username sc-servicename s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs-bytes sc-bytes time-taken cs-version cs(User-Agent) cs(Cookie) cs(Referer)
2012-08-16 11:55:13 172.20.1.0 - HTTP 192.168.6.254 8080 GET /Citrix/XenApp/Wan/auth/login.jsp - 302 247 355 1 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+Trident/4.0;+.NET+CLR+1.1.4322;+.NET+CLR+2.0.50727;+.NET+CLR+3.0.04506.648;+.NET+CLR+3.5.21022) - -
misc/log-analytics/tests/logs/nginx_json.log supprimé 100644 → 0
+ 0
− 2
Voir le fichier @ 74f0d0ff
{"idsite":1,"ip": "203.38.78.246","host": "www.piwik.org","path": "/piwik.php?action_name=Clearcode%20-%20Web%20and%20Mobile%20Development%20%7C%20Technology%20With%20Passion&idsite=1&rec=1&r=983420&h=17&m=31&s=25&url=http%3A%2F%2Fclearcode.cc%2F&urlref=http%3A%2F%2Fclearcode.cc%2Fwelcome&_id=1da79fc743e8bcc4&_idts=1360047661&_idvc=1&_idn=0&_refts=1360047661&_viewts=1360047661&_ref=http%3A%2F%2Fpiwik.org%2Fthank-you-all%2F&pdf=1&qt=1&realp=0&wma=1&dir=1&fla=1&java=1&gears=0&ag=1&cookie=1&res=1680x1050","status": "200","referrer": "http://clearcode.cc/","user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17","length": 192,"generation_time_milli": 0.008,"date": "2013-10-10T16:52:00+02:00"}
{"idsite":1,"ip": "203.38.78.246","host": "www.piwik.org","path": "/piwik.php?action_name=AdviserBrief%20-%20Track%20Your%20Investments%20and%20Plan%20Financial%20Future%20%7C%20Clearcode&idsite=1&rec=1&r=109464&h=17&m=31&s=40&url=http%3A%2F%2Fclearcode.cc%2Fcase%2Fadviserbrief-track-your-investments-and-plan-financial-future%2F&urlref=http%3A%2F%2Fclearcode.cc%2Fwelcome&_id=1da79fc743e8bcc4&_idts=1360047661&_idvc=1&_idn=0&_refts=1360047661&_viewts=1360047661&_ref=http%3A%2F%2Fpiwik.org%2Fthank-you-all%2F&pdf=1&qt=1&realp=0&wma=1&dir=1&fla=1&java=1&gears=0&ag=1&cookie=1&res=1680x1050","status": "200","referrer": "http://clearcode.cc/","user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17","length": 192,"generation_time_milli": 0.008,"date": "2013-10-10T16:52:00+02:00"}
misc/log-analytics/tests/logs/s3.log supprimé 100644 → 0
+ 0
− 1
Voir le fichier @ 74f0d0ff
b659b576cff1e15e4c0313ff8930fba9f53e6794567f5c60dab3abf2f8dfb6cc www.example.com [10/Feb/2012:16:42:07 -0500] 1.2.3.4 - EB3502676500C6BE WEBSITE.GET.OBJECT index "GET /index HTTP/1.1" 200 - 368 368 10 9 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
misc/log-analytics/tests/run_tests.sh supprimé 100755 → 0
+ 0
− 8
Voir le fichier @ 74f0d0ff
#!/bin/sh
cd $(dirname $0)
# Make sure nosetests is installed.
nosetests -V >/dev/null 2>&1 || (echo "nose (http://readthedocs.org/docs/nose/en/latest/) must be installed"; exit 1)
PYTHONPATH=.. nosetests tests.py $*
misc/log-analytics/tests/tests.py supprimé 100644 → 0
+ 0
− 693
Voir le fichier @ 74f0d0ff
Ce diff est replié.
0% ou .
You are about to add 0 people to the discussion. Proceed with caution.
Veuillez vous inscrire ou vous pour commenter