Upgrading

This is a patch release for 1.14.5 with no schema change. To upgrade, replace the files. php cli.php cmd=update has nothing to apply.

Fixes

• The Last seven days period covered eight days: today and the seven days before it. It now covers today and the six days before it.
• The Last week period ended on the Sunday after the week instead of on the Saturday.
• Every relative period (Yesterday, Last week, Last thirty days and the others) is now counted in calendar days instead of seconds, so a daylight-saving change no longer shifts one by a day.

These are fixed in #1305.

Upgrading

A patch release on 1.14.4, with no schema change: replace the files. php cli.php cmd=update has nothing to apply.

Fixes

• A trend chart shows the last day of its period. It returned one day fewer than the period held, so a chart ending today left today off (#1302).

Upgrading

A patch release on 1.14.3, with no schema change: replace the files. php cli.php cmd=update has nothing to apply.

Fixes

• A heatmap draws its clicks again. Since 1.13.0 its data request was refused, so the overlay opened on the page with nothing on it; 1.14.3 fixed only the link that opens it (#1297).
• The heatmap link no longer fails when the site is not already cached (#1295).

Upgrading

A patch release on 1.14.2, with no schema change: replace the files. php cli.php cmd=update has nothing to apply.

Fixes

• The Heatmap Overlay link opens the page again. Since 1.10.1 it went to the dashboard instead (#1293).
• The heatmap opens only a page on the site's own domain, for a user who may see the site (#1293).

Upgrading

A patch release on 1.14.1, with no schema change: replace the files. php cli.php cmd=update has nothing to apply.

A page whose URL was cut short by the query-filter bug (#895) is recorded under its full URL from now on, separately from its history.

Fixes

• Filtering a query parameter out of a page URL no longer drops the parameters after it (#895).
• A regex goal condition may contain an @ (#909).
• Behind a reverse proxy on a sub-path, signing in returns to the page you asked for (#863).
• owa_cmds.push(['OWA.…', …]) no longer creates a stray tracker named OWA (#858).
• Script tags no longer carry type="text/javascript" (#916).

Thanks

@mpez7, @wernight, @snailwei, @ivan-docflo and @DrOuille9 for the reports.

Upgrading

A patch release on 1.14.0, with no schema change (schema 33). Replace the files; php cli.php cmd=update has nothing to apply. No new cron entry.

Recommended for any installation tracking more than one site where some users are granted only some of them.

Changes

A report of one record reads it only from the site the request names. The clickstream, transaction and transaction_items reports of the REST API, and the recording player, looked a session, order or recording up by its id alone. Access is checked against the site a request names, so a user granted one site could read another site's record by naming their own. They now return a record only when it belongs to that site, as the other reports already did.

This release contains that fix and nothing else from master.

Thanks

@HieuPenguinnn for reporting the issue (#1236).

Upgrading

Schema 27 → 33. Run php cli.php cmd=update. No new cron entry.

Update 031 rewrites stored (not set) values to NULL. It runs as part of the upgrade rather than on request, because leaving some rows with the string and others with NULL would show one missing value as two separate values in a report.

Run both of these after upgrading

Each repairs data that is wrong or missing today. They are commands rather than upgrade steps because both walk the whole table and join across several others — on a database shared with other sites that is your call to time, not something an upgrade should do to you while you are not looking. Both are safe to re-run and safe to interrupt, and both take a dry run first.

php cli.php cmd=backfill-visitor-acquisition

Acquisition was not recorded before this release, so every visitor you already have has none — and the five new Acquisition dimensions will show almost nothing until this runs. It recovers the values from the session each visitor first arrived in. Skips rows already filled, so an interrupted run picks up where it stopped. Takes --dry-run and batch=N.

php cli.php cmd=repair-geo-encoding

City and region names containing accented characters are already stored wrong, and display as mojibake in your reports. The geolocation reader converted names from ISO-8859-1 to UTF-8, but MaxMind stores them as UTF-8 already, so they were written double-encoded. This rewrites the affected rows. Takes dry-run=1.

Changes

How a visitor was acquired is now recorded, and reportable. The source, medium, campaign, ad and search terms of the visit that first brought someone to the site are written onto the visitor when they are created, and never overwritten by a later campaign. Five dimensions expose it — Acquisition Source, Medium, Campaign, Ad and Search Terms — so you can group any visitor-scoped report by where those people originally came from, rather than by the campaign that happened to bring them back. Visitors you already have need the backfill above before any of this shows up.

Custom reports can be shared. A report is visible only to its author until it is listed to everyone. Readers can star reports and visualizations to keep them to hand.

Cookie lifetimes are configurable from the tracking snippet — visitor, session and campaign, in days, where they were previously fixed at 364, 364 and 60. The cookie_persistence setting also reaches the JavaScript tracker now. It has made server-set cookies session-only since 2016, but the tracker never read it, so an installation with persistence turned off was still handing out a year-long visitor cookie from every page.

An unrecognised snippet command no longer stops tracking. Browsers cache the tracker for days, so a snippet generated by newer PHP can push a command the visitor's cached tracker does not know. That threw, and abandoned every command still queued behind it. Since the snippet pushes trackPageView last, those visits were not recorded at all, with nothing in any log to say so. Unknown commands are now skipped.

Absent tracking values are stored as NULL instead of the string (not set), which sorted among real values, drew its own slice in a pie chart, and could not be told apart from a visitor who genuinely typed it.

Feed requests are stored as events, filed under yyyymmdd like the rest.

Fixed: a fatal error in the MaxMind web-service lookup path, which took down tracking on installs using it rather than the local database.

Removed: six owa_visitor columns holding a partial copy of each visitor's last session. Nothing has written them for years and no report reads them, so no report changes.

Contributors

@digikwal — configurable cookie lifetimes and the command-queue fix (#1098), geolocation name encoding (#1100), and the MaxMind web-service fatal (#1101).

OWA 1.13.0 is a major release that sets the stage for the upcoming 2.0 branch of rleases. As such, there are multiple changes to the admin interface, a renovation of existing features and code base, as well as several exciting new features (custom reports, etc.).

Read the upgrade notes before istalling as there are multiple database schema updates and changes to OWA admin urls. While it is a large release with many moving parts, there should be no breaking changes.

See the wiki for updated documentation.

Upgrading

1. Apply the schema updates. This release moves the base schema from 16 to
27:

php cli.php cmd=update

Nothing periodic runs until updates complete — the job scheduler refuses every job while updates are pending. The updates create the organization, property, setting, notification, custom report and goal event tables, and migrate existing goals into them. They are idempotent, so a run that stops part way can be run again.

2. Report URLs have changed. Reports are now reached through one action with a report id, rather than an action per report:

before:  index.php?do=base.reportPages&siteId=...
now:     index.php?do=base.report&reportId=pages&siteId=...

There is no redirect from the old form — an old link answers 404. Anything outside OWA that points at a report URL (bookmarks, dashboards, emailed links) needs updating. Links inside OWA are already correct. The report ids are the filenames in modules/Base/reports/.

3. Some reports are gone. Eight detail reports whose parent report already showed the same thing were retired, along with the transaction detail report. Those ids answer 404 rather than redirecting.

4. cmd=activate now installs. It creates a module's tables and records its schema version, which is what the Modules screen's Activate control has always done — previously the command only set the active flag, leaving a module switched on with no tables. cmd=install-module is deprecated and will be removed in 2.0; it still works and does the same thing.

If a module was enabled with the old command, php cli.php cmd=instance-info names it as "active, but never installed", and running cmd=activate on it repairs it.

Existing site ids do not change. Site identity is now minted rather than derived from the domain, but no update rewrites existing values — tracking snippets already deployed keep working.

Check the result:

php cli.php cmd=instance-info

New in this release: one report covering the environment, module and schema state, the scheduler, partitioning, data freshness, queue depth and what the instance holds.

New

• Reports are configuration (#1044, #1047). Around sixty reports are now definitions rather than controllers, and users can build their own custom reports from the same pieces.
• Organization, Property and Observation Profile (#1063). A tracked website and the way it is observed are separate things, and settings live at the tier they belong to.
• Goal events (#1064). Goals were twenty numbered slots per site; they are now rows, modelled for what v2 needs.
• cmd=instance-info (#1082). The whole instance in one report, safe to run at any time.
• A signed-in user can edit their own account (#1070).
• Dimension catalog keyed by entity, and scopable (#1049); metrics are declarations (#1053); the tracking wire surface is enumerated in one config file (#1055, #1058).
• Campaign tags are named apart from the source they resolve to (#1056).
• Timezone is asked for at install (#1043), because it is baked into every fact row and cannot be changed retroactively.
• Site identity is minted, not derived from the domain (#1050), so one domain can be measured by more than one observation profile/tracker.

Fixed

• A tracking request could overwrite values the server derives (#1054). A parameter whose name matched a column was written to that column, including the observed IP address and the event timestamp.
• Schema updates are idempotent in both directions (#1065), and addColumn() now works for an indexed column (#1066)
• Activating a module no longer skips its updates (#1083). Re-running an install advanced the schema version without migrating the tables, permanently skipping everything in between.
• Installer now checks for build packages (#1081) instead of answering every request with a blank 500.
• The installer no longer loads the config file it just wrote (#1072).
• An unset column is written as NULL, not its type's zero (#1045).
• A mistyped password no longer ends the password reset (#1059), and the login and reset screens no longer say two things they shouldn't (#1060).
• doAction() no longer discards what errorAction() returned (#1071), which blanked the page on the most ordinary form mistakes.
• userName is a column on the fact row, not a table to join (#1051).

Also

• Three vulnerable build-toolchain dependencies patched (#1075).
• The schema updates are now actually exercised by CI against a database with a past, data included (#1067, #1068).
• Migration and modernization leftovers removed (#1048, #1074, #1076).

Overview

This is a bug fix and maintenance release that ships with an optional maintenance CLI command for improving key generation. The optional CLI step is essential for long term data quality on large installs.

Upgrading

Schema 15 → 16. Run cmd=update (or visit the admin UI, which prompts). The update writes a single setting and takes no locks, so it does not require CLI mode.

One optional manual step, at your convenience. Dimension ids are 63-bit from this version on. A new installation gets that immediately. An existing installation keeps deriving the old 32-bit ids until you run:

php cli.php cmd=rederive-dimension-ids

Ids stay internally consistent either way, so there is no rush and nothing breaks if you never run it — but until you do, the installation carries use_32bit_hash and the wider id space is not in use. The migration is deliberately a command you run when it suits you rather than something an upgrade does to you.

Partition your fact tables first — this is a prerequisite, not a tuning tip. The rewrite is issued one partition at a time; on an unpartitioned table it becomes one unbounded statement per key column across the whole table, holding a single long lock while tracking continues. There is no hurry to convert ids, so partition first and convert afterwards.

Read Converting Dimension Ids before you start — it covers the preparation, timing and performance (about 8 minutes for 136,000 ids and 2.5 million fact rows on storage sustaining 3,000 IOPS; the work is I/O-bound, so sustained IOPS decides the duration), and what the conversion deliberately leaves behind.

No new cron entry. The scheduler's single entry from 1.11.0 is unchanged.

Security

Domstreamand heatmap playback The overlay uses a new short lived token and no longer relies on the overlay cookie.

JSONP is gone, client and server. the two cross site overlays that needed it now fetch over CORS.

Tracking events for unregistered sites are refused. Your site_id must be the in the owa_sites table prior to the recording of tracking events.

Fixed

• CORS requests addCorsHeaders() compared site row arrays against the Origin string, so no Access-Control-Allow-Origin was ever sent. Origins now match on host, so an install storing http://example.com while serving over https still works. Vary: Origin is always sent.
• The heatmap reported fixed Overlay API links lost their siteId; the player failed loudly with a 422 but the heatmap returned 200 for a query with no site filter and a defaulted period.
• Event queues could stop draining permanently. Messages queued before the PSR-4 relocation named the pre-namespace class and could not be decoded properly.
• Site access UI The UI arround adding per site permissions has been totally reworked to avoid accidental user errors.
• A refused database connection took down the whole request instead of degrading.
• A robot token at the start of a user agent was not counted as a robot.
• Asset URLs were built with a doubled slash.
• Partition-rotate's refusal message read "no fact table is not partitioned".

Changed

• Two goal types are removed. visit_duration and pages_per_visit were never reachable — the dispatcher sent visit_duration to the pages-per-visit checker, neither npvs nor session_duration was ever written, and both switches lacked a break. Only url_destination was ever functional. Existing goals of the removed types were already inert.

OWA 1.11.0 adds two subsystems that work together: date partitioning for the fact tables, and a job scheduler that keeps them maintained from a single cron entry.

Upgrading: three things to do

1. Apply the database updates. The required schema version moves from 12 to 15. OWA prompts you in the browser, or:

/path/to/php cli.php cmd=update

Until this runs, the scheduler refuses every job and says so.

2. Add the cron entry. This is new, and nothing periodic runs without it:

* * * * * cd /path/to/owa && php cli.php cmd=schedule-run

One line, whatever you schedule later. Because a scheduler that is not running produces no output, OWA reminds you at the top of every admin page until the entry exists. Check it took with cli.php cmd=schedule-status.

3. Run partition-init when you are ready. New installations are partitioned automatically. An existing installation is deliberately not converted during the update, so partitioning does nothing for you until you convert it yourself:

/path/to/php cli.php cmd=partition-init --dry-run   # see the plan, change nothing
/path/to/php cli.php cmd=partition-init             # convert

Run it in a maintenance window. It rewrites each fact table twice, once to widen the primary key and once to apply the partitioning, and MySQL blocks writes on a table for the duration. Budget roughly five seconds per million rows, per table. That is why it is not part of the update: an upgrade should not decide for you when your tracking stops accepting writes.

There is no hurry and nothing degrades while you wait. Until you convert, the rotate-partitions job refuses with "no fact table is partitioned", which is the expected state rather than a fault. What you are waiting for is the payoff: retention becomes a metadata operation instead of a DELETE over millions of rows, and reports read only the periods covering the reporting period.

See Partitioning Fact Tables for what the layout looks like, how granularity is chosen, and what the conversion does.

Partitioning the fact tables (#989, #994)

All seven fact tables are split into periods on their yyyymmdd column. Dropping old data becomes a metadata operation instead of a DELETE over millions of rows, and reports read only the periods covering the reporting period.

• Tiered layouts. Recent months keep full granularity; older history is merged into whole years, and into runs of years if needed, so a long-running installation stays inside the server's open-file budget. Controlled by OWA_PARTITION_DETAIL_MONTHS (36).
• Nothing is deleted by default. partition-rotate ships with no keep, so it maintains the twelve-month lead and merges the tail without discarding data. Retention is opt-in, via OWA_SCHEDULED_JOBS.
• Four commands: partition-status (read-only), partition-init (one-time conversion), partition-rotate (maintenance), partition-reorganize (change granularity), partition-drop (remove periods past a cutoff).

A job scheduler (#997, #1000)

Periodic work is registered in code and dispatched from one cron entry, rather than an entry per task.

• One registered job: rotate-partitions, monthly, running partition-rotate with no arguments.
• Configurable per installation through OWA_SCHEDULED_JOBS in owa-config.php: retune what ships, add jobs, or turn one off. Overrides are per key, so changing arguments keeps the shipped schedule.
• A missed job runs once, not once per miss. The scheduler records the occurrence a run satisfied, so a machine that was off for three days runs a daily job once when it comes back.
• Jobs never overlap themselves, via a per-job lock.
• schedule-status explains why a job is not running, working through the possible causes and naming the first that applies rather than reporting only that something is behind.

Fixes

• Password reset returned a 500 on PHP 8, and never sent mail when a From address was configured (#990, @digikwal)
• Visitors roster showed the wrong day (#991)
• JS bundles leaked globals; they are now wrapped in an IIFE (#993, @digikwal)
• Asset URLs were built with a doubled slash (#998)
• A missing site id, and an unresolvable action, are now answered rather than raising (#983, #996)
• addIndex() no longer adds duplicate copies of an index, and exact duplicate indexes are dropped (#985, #986)
• visitor_id is now indexed on the fact tables (#988)
• The error log is created group-writable (#987)
• CLI boolean arguments accept --switch syntax (#995)
• Pinned patched js-yaml and fast-uri through overrides (#984)

Documentation

New wiki pages: Scheduled Jobs and Partitioning Fact Tables. Installation, Updating and Technical Requirements now cover the cron entry.

Contributors

Thank you to @digikwal for the PHP 8 password-reset fix and mailer From handling (#990), and for wrapping the JS bundles so they stop leaking globals (#993).

Internal

Test and CI work, not user facing: an isolation sweep that runs every test file in its own process, and the defects it found — settings flushed from a destructor could be lost and could exit the process 255, and Mysql::close() was not idempotent (#1001, #1002, #1003).