Note the following limitations of hive notification:
Arcadia Enterprise connects to the MQ server, and uses the Hive notification system when it is available. Otherwise, it proceeds with its normal operation without notifications.
ArcViz makes attempts to connect to MQ server port every 60 seconds, if it is not
connected already. After it finally connects to the MQ server after a period of not being
able to listen to notifications, it may have stale data. This situation may require a
manual invalidate metadata
command.
When files are loaded directly into an existing HDFS directory that corresponds to an
existing table or table partition, there is no notification. You must issue a manual
invalidate metadata
command on that table to see the changes.
Hive Notification has the following default behaviors:
Because Hive does not change any metadata when it is down, we don't worry about missed notifications. However, we can run into a situation where ArcEngine or Impala cannot be reached, so we queue messages, and then Hive is restarted. In such cases, the user must manually refresh or invalidate the metadata on ArcEngine or Impala.
If Impala or ArcEngine are down, we queue notifications so they may be processed as
refresh table_name
or invalidate table_name
after these services come up again, so they have the most recent copy of metadata.
In order to implement the alter table rename
and alter table change
column
SQL calls, ArcEngine invalidates all analytical views associated with the
altered table.
invalidate analytical view table table_name;