When you enable the Hive Notification feature, all changes made in Hive propagate automatically to the ArcEngine.
Consider the following two scenarios:
This behavior is in place at all times, even with Hive Notification disabled.
This behavior is only possible when you explicitly enable the notification feature.
To enable this feature, perform the following tasks:
To review the possible effect of Hive notification on various DDL and DML changes, see Notification Types.
To understand the limitations of this feature, see Limitations of the Hive Notification Mechanism.
You must install an active Message Queue (MQ) server that listens for relevant notifications and routes them to ArcEngine.
Follow these steps to set up the MQ server:
arcadia
user.Download the activemq
MQ files from the following URL: http://activemq.apache.org/activemq-5153-release.html.
This site offers both Windows and Unix/Linux/Cygwin distributions.
Unzip the downloaded file.
61616
, by
default:> bin/activemq start
After you successfully install the MQ server, you must configure Hive to point to it.
Please contact support@arcadiadata.com for help with this task.
In the Arcadia Catalog Cache Advanced Configuration Snippet (Safety
Valve) for flagfile
, add the following
information:
arcadia_hive_notify_mq_host=localhost
arcadia_hive_notify_mq_port=61616
arcadia_invalidate_metadata_on_restart=true
In Ambari Stacks, set the following parameters:
localhost
. 61616
.arcadia_invalidate_metadata_on_restart=true
.Automatic Hive Notification supports the following types of events:
A CREATE TABLE
statement produces a notification to ArcEngine as an
invalidate metadata table_name
command.
DROP TABLE
statement produces a notification to ArcEngine as an
invalidate metadata table_name
command. Additionally, any
analytical view that use the table becomes invalid after a query attempts to use it.When LOAD
ing data to a table, either as rows or as an entire partition, or when
INSERT
ing rows into a table, the notification to ArcEngine appears as
a refresh table_name
command. The next time ArcEngine routes a
query to analytical views based on that table, it marks the views as stale, unless the
analytical views refresh before then, through a manual or scheduled refresh.
Depending on the nature of the ALTER TABLE
statement, there may or may not be an
impact on ArcEngine. Some of the following statements only affect the metadata.
The RENAME table_name
clause is
equivalent to creating a new table, and issuing an invalidate
metadata
command. This affects not just analytical views, but also
datasets and any dependant datasets and apps. Arcadia Enterprise does not support ALTER TABLE
RENAME
command.
Some table properties affect ArcEngine.
This clause deals with processing data, and does not affect ArcEngine.
This clause does not affect the directories where the data resides, so does not affect ArcEngine.
This clause applies to future storage of data, so it has no effect on ArcEngine.
Similarly, this clause does not affect the directories where the data resides, only the future data, so it can be ignored.
Does not affect directories where the data resides, only the future data, so it can be ignored.
ADD CONSTRAINT or DROP CONSTRAINT
These clauses do not affect data, and may be ignored.
The change (column_name)
clause results in an
invalidate metadata table_name
notification. This
notification also leads to invalidating analytical views associated with the entire
table, not just the column.
ADD PARTITION, DROP PARTITION, or PARTITION ... RENAME TO
These clauses generate notifications to refresh table_name
.
The next time ArcEngine routes a query to analytical views based on that table, it
marks the views as stale, unless the analytical views refresh before then, through a
manual or scheduled refresh.
RECOVER PARTITIONS or MSCK REPAIR TABLE
These clauses require a change to the metadata, and result in a refresh
table_name
notification to ArcEngine.
ARCHIVE PARTITION or UNARCHIVE PARTITION
This clause removes files, and therefore affects data. It results in a
refresh table_name
notification to ArcEngine.
This clause does not affect data, and so it does not issue notifications..
This clause does not affect data, and so it does not issue notifications.
ENABLE NO_DROP or DISABLE NO_DROP
This clause does not affect data, and so it does not issue notifications.
ENABLE or DISABLE OFFLINE
This clause send a refresh table_name
notification to
ArcEngine.
This clause does not affect data, and so it does not issue notifications.
This clause does not affect data, and so it does not issue notifications.