ARMY MARKETING
The Army Has Taken
Recent Actions That
Could Improve
Program Oversight,
Effectiveness, and
Workforce Practices
Report to Congressional Committees
November 2019
GAO-20-93
United States Government Accountability Office
______________________________________ United States Government Accountability Office
November 2019
ARMY MARKETING
The Army Has Taken
Recent Actions That Could
Improve
Program Oversight, Effectiveness, and
Workforce
Practices
What GAO Found
The Army has recently taken steps to improve the oversight of its primary
marketing and advertising contract and measurement of the effectiveness of its
marketing and advertising program in response to two U.S. Army Audit Agency
(AAA) reports. In April 2018, AAA found that the Army Marketing and Research
Group (AMRG)—the component responsible for conducting the Army’s national-
level marketing and advertising program—did not fully evaluate the performance
of its contracted advertising agency or track the effectiveness of its marketing
and advertising efforts. GAO found that AMRG has taken or is taking actions to
address AAA’s recommendations:
• Contract Oversight. AMRG has developed processes for overseeing the
advertising agency’s performance and services. For example, AMRG
developed a form program managers use to validate that proposed
advertising services are not already provided through other contracts.
• Program Effectiveness. AMRG has taken steps in several areas related to
revising its strategic marketing goals to support Army recruiting, updating
how it assesses marketing and advertising effectiveness, and improving the
reliability of data systems. AMRG’s steps are consistent with commercial
best practices that GAO identified for assessing the effectiveness of
advertising, such as identifying outcomes from advertising.
Examples of Army Marketing and Advertising Activities, Including Promotional Items, Website,
and Mobile Assets
The Army has also taken steps to improve the workforce practices and
organizational structure of its marketing and advertising program in response to
two workforce reviews. The two reviews—by an Army organization and the U.S.
Office of Personnel Management (OPM)—found that AMRG, among other things,
did not have regular communication throughout its workforce and with its
stakeholders, and had a poor workforce climate. AMRG took initial steps to
address the reviews’ recommendations. The Army then established a new office
effective August 2019—the Office of the Chief Army Enterprise Marketing—to
replace AMRG and to assume all marketing and advertising activities. Some of
the Army’s early steps to establish the new office are consistent with key
practices for agency reform efforts identified previously by GAO. For example,
the Army outlined a three-phased plan with specific tasks and associated dates
to fully establish the new office by early 2020 consistent with the key practice to
establish implementation goals and a timeline.
Why GAO Did This Study
The Army requested nearly $335 million
for fiscal year 2020 to conduct
marketing and advertising activities
intended to increase awareness of Army
service
and ultimately generate leads for
potential recruits.
In April 2018, AAA made
recommendations in two reports to
improve the contract oversight and
return on investment of the Army’s
marketing and advertising program.
Further, in May 2018 and October 2018,
respectively, the Army and OPM made
recommendations to improve the
workforce practices and organizational
structure of the program.
The John S. McCain National Defense
Authorization Act for Fiscal Year 2019
included a provision for GAO to review
the actions taken to implement AAA’s
recommendations and the effects of
these actions on AMRG’s leadership,
workforce and business practices, and
return on investment.
This report assesses the extent to which
the Army has taken steps to address
recommendations (1) from AAA to
improve the contract oversight and
measurement of the effectiveness of the
Army’s marketing and advertising
program and (2) from the Army and
OPM to improve the workforce practices
and organizational structure of the
marketing and advertising program.
GAO analyzed Army marketing and
advertising data from fiscal year 2018;
reviewed marketing and advertising
plans and guidance; conducted focus
groups with AMRG personnel;
interviewed cognizant officials; and
compared the Army’s efforts to GAO-
identified best practices.
The Army provided technical comments,
which GAO incorporated as appropriate.
View GAO-20-93. For more information, contact
Elizabeth Field at (202) 512
-2775 or
fielde1@gao.gov
.
Highlights of GAO-20-93, a report to
congressional committees
Page i GAO-20-93 Army Marketing
Letter 1
Background 6
The Army Has Taken Steps to Develop Processes to Improve Its
Oversight of Its Primary Marketing and Advertising Contract 11
The Army Has Taken Steps to Improve Its Approach for
Measuring the Effectiveness of Its Marketing and Advertising
Program 17
The Army Has Taken Steps to Improve Its Workforce Practices
and to Reorganize the Organizational Structure of Its Marketing
and Advertising Program 26
Agency Comments 31
Appendix I Army Actions to Improve Its Marketing and Advertising Program
as Stated in the Army’s Report to Congress 33
Appendix II Army Marketing and Research Group’s Reported Actions
to Address Army Audit Agency’s Recommendations 35
Appendix III GAO Contact and Staff Acknowledgments 40
Tables
Table 1: Army Organizations with Roles and Responsibilities in
the Army’s Marketing and Advertising Program 7
Table 2: Status of the Army’s Implementation of U.S. Army Audit
Agency (AAA) Recommendations to Improve Contract
Oversight within the Marketing and Advertising Program 12
Table 3: Status of the Army’s Implementation of U.S. Army Audit
Agency (AAA) Recommendations to Improve the Return
on Investment of the Marketing and Advertising Program 18
Table 4: The Army’s Reported Actions to Congress on Improving
the Army’s Marketing and Advertising Program 33
Table 5: Army Marketing and Research Group’s (AMRG) Actions
to Address U.S. Army Audit Agency (AAA)
Recommendations on Contract Oversight 36
Contents
Page ii GAO-20-93 Army Marketing
Table 6: Army Marketing and Research Group’s (AMRG) Actions
to Address U.S. Army Audit Agency (AAA)
Recommendations on Return on Investment 38
Figures
Figure 1: Examples of Army Marketing and Advertising 9
Figure 2: Army Marketing and Advertising Phases and Activities in
Support of Recruitment Goals 10
Figure 3: Comparison of the Fiscal Year 2018 and Fiscal Year
2019 Army Marketing Goals to Marketing and Advertising
Phases 20
Abbreviations
AAA U.S. Army Audit Agency
AEMO Office of the Chief Army Enterprise Marketing
AMRG Army Marketing and Research Group
ASA(ALT) Assistant Secretary of the Army for Acquisitions,
Logistics, and Technology
DOD Department of Defense
EMM Enterprise Marketing Management
NDAA National Defense Authorization Act
OPM U.S. Office of Personnel Management
USACC U.S. Army Cadet Command
USAREC U.S. Army Recruiting Command
This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.
Page 1 GAO-20-93 Army Marketing
441 G St. N.W.
Washington, DC 20548
November 18, 2019
The Honorable James M. Inhofe
Chairman
The Honorable Jack Reed
Ranking Member
Committee on Armed Services
United States Senate
The Honorable Adam Smith
Chairman
The Honorable Mac Thornberry
Ranking Member
Committee on Armed Services
House of Representatives
Each year, the Army seeks to achieve its recruitment goals by targeting
the small percentage of the U.S. population able to meet the education,
physical fitness, and age requirements for entry into military service. The
Army did not meet its recruitment goals in fiscal year 2018 and reported
meeting them in fiscal year 2019. To help achieve these goals, the Army
requested nearly $335 million for fiscal year 2020—an increase of about
31 percent over the prior year—to conduct marketing and advertising
activities. These activities include television and radio advertisements,
internet and other digital marketing, and direct mail to potential recruits.
1
Over the past few years, we and other organizations have identified
challenges within the Army’s marketing and advertising program and
recommended actions to improve the program’s operations. In May 2016,
we found that, among other things, the military services, including the
Army, did not fully measure the effectiveness of their advertising
activities.
2
We recommended that the Department of Defense (DOD)
ensure that the military services fully measure the performance of
1
According to Department of Defense (DOD) Instruction 1304.35, marketing is the
systemic planning, implementation, and control of a mix of activities intended to facilitate
joining a military service, and advertising is defined as a subset of marketing and is the
paid or unpaid persuasive message to the public that promotes military service. DOD
Instruction 1304.35, Military Marketing (Nov. 1, 2017).
2
GAO, DOD Advertising: Better Coordination, Performance Measurement, and Oversight
Needed to Help Meet Recruitment Goals, GAO-16-396 (Washington, D.C.: May 12, 2016).
Letter
Page 2 GAO-20-93 Army Marketing
advertising activities. DOD partially concurred with this recommendation.
3
DOD has taken steps to address this recommendation by issuing
guidance requiring the military services to establish metrics, including
measurable goals, by which the success of a service or component’s
overall marketing plan or events can be measured.
4
Subsequent to our report, organizations within and outside of the Army
have reviewed the Army’s marketing and advertising program and made
recommendations to improve its operations. In two reports issued in April
2018, the U.S. Army Audit Agency (AAA) found that the Army Marketing
and Research Group (AMRG)
5
—the Army component that was
responsible for conducting the Army’s national-level marketing and
advertising program—could improve its processes and procedures to
demonstrate that its activities provided what AAA termed a positive
“return on investment” and to increase management controls and
oversight of its marketing and advertising contracts.
6
AAA made seven
recommendations in each of the two reports, and the Army concurred
with all 14 recommendations.
In addition, in May 2018, the Office of the Assistant Secretary of the Army
for Acquisitions, Logistics, and Technology (ASA(ALT)) identified several
high-risk issues that contributed to organizational inefficiencies within
AMRG and made recommendations to improve performance
transparency, mission-to-function alignment, and program accountability.
7
3
DOD agreed with this recommendation in broad terms and stated that the military
services should include measurable goals in their advertising plans and have access to
necessary performance data to measure effectiveness.
4
DOD Instruction 1304.35.
5
As described later in this report, effective August 1, 2019, the Army redesignated AMRG
as the Office of the Chief Army Enterprise Marketing (AEMO). Because we conducted
most of this work prior to August 2019, we primarily refer throughout this report to AMRG.
This includes attributing information to officials as being from AMRG even after they had
been transferred to the Office of the Assistant Secretary of the Army for Manpower and
Reserve Affairs.
6
AAA, The Army’s Marketing and Advertising Program: Return on Investment, Audit
Report A-2018-0036-MTH (Fort Belvoir, Va.: Apr. 13, 2018) and The Army’s Marketing
and Advertising Program: Contract Oversight, Audit Report A-2018-0033-MTH (Fort
Belvoir, Va.: Apr. 13, 2018).
7
Office of the Deputy Assistant Secretary of the Army for Plans, Programs, and
Resources, Assistant Secretary of the Army for Acquisitions, Logistics, and Technology,
Army Marketing and Research Group: Business and Process Improvements Initiative
(May 2018).
Page 3 GAO-20-93 Army Marketing
Further, in accordance with section 599 of the John S. McCain National
Defense Authorization Act for Fiscal Year 2019 (hereafter NDAA for
Fiscal Year 2019), the Army reported that AMRG requested support from
the U.S. Office of Personnel Management (OPM) in reviewing AMRG’s
organization.
8
In October 2018, OPM issued a study identifying
organizational design issues contributing to challenges with AMRG’s
reputation, trustworthiness, work environment, and organizational
performance, and identified actions that AMRG could take to improve its
workforce practices.
9
Section 599 of the NDAA for Fiscal Year 2019 included a provision that
we conduct a review of the actions taken to implement AAA’s
recommendations and of the effects of these actions on AMRG
leadership, workforce and business practices, and return on investment.
10
In this report, we assess the Army’s response to recommendations from
AAA, ASA(ALT), and OPM. Specifically, we assess the extent to which
the Army has taken steps to: (1) improve contract oversight within its
marketing and advertising program; (2) improve how AMRG measures
the effectiveness of its marketing and advertising program; and (3)
improve the workforce practices and organizational structure of the
marketing and advertising program. In appendix II, we provide a
consolidated summary of the 14 recommendations from the AAA reports
on contract oversight and return on investment, the actions that AMRG
has taken to address them, and the status of their implementation,
according to AAA, as of September 2019.
For the first objective, we reviewed documentation and interviewed
AMRG, AAA, and U.S. Army Mission Installation and Contracting
8
Section 599 of the NDAA for Fiscal Year 2019 required AMRG, in consultation with OPM,
to assess AMRG’s organizational structure, staffing, and training, including an assessment
of the workplace climate and culture internal to AMRG. Pub. L. No. 115-232, § 599 (2018).
9
OPM, Report of Findings and Recommendations: Organizational Design Study (October
2018).
10
Pub. L. No. 115-232, § 599 (2018). Section 599 of the NDAA for Fiscal Year 2019 also
required the Army to submit a report to Congress that addressed several elements, such
as the mitigation and oversight measures implemented to assure improved program return
and contract management and the establishment of a review process to regularly evaluate
the effectiveness and efficiency of marketing efforts. The Army submitted the required
report on February 5, 2019. In appendix I, we identify the required elements and actions
the Army has reported taking in response.
Page 4 GAO-20-93 Army Marketing
Command officials on AMRG’s actions to address deficiencies identified
in the AAA report on contract oversight. We analyzed the number of task
orders issued and amounts obligated from fiscal year 2011 through 2018
on AMRG’s primary marketing and advertising contract.
11
We obtained
this data from the Federal Procurement Data System-Next Generation,
which is a primary source of federal procurement data. We chose data
starting in fiscal year 2011 because the Army awarded the previous
contract in March 2011 and ending at the end of fiscal year 2018 because
it was the last complete fiscal year of data at the time of our review. To
assess the reliability of the Federal Procurement Data System-Next
Generation data that we used, we reviewed the data dictionary, data
validation rules, data outputs, and other documentation. Based on these
steps, we determined that the data were sufficiently reliable for the
purposes of identifying the number of task orders issued and amounts
obligated on AMRG’s primary marketing and advertising contract.
For the second objective, we reviewed documentation on the Army’s
process for developing its annual marketing plan, goals, and performance
measures, and we conducted interviews with officials at organizations
with responsibilities for implementing the Army’s marketing and
advertising program, including AMRG; the U.S. Army Training and
Doctrine Command (TRADOC); Office of the Chief, Army Reserve; and
Army National Guard. We also conducted a site visit to Fort Knox,
Kentucky, to meet with officials from U.S. Army Recruiting Command
(USAREC), U.S. Army Cadet Command (USACC), and the U.S. Army
Marketing and Engagement Brigade to discuss their efforts to measure
the effectiveness of their marketing and advertising activities and to
observe Army marketing and advertising facilities and assets. We
compared this information with the three categories of commercial best
practices for assessing the effectiveness of advertising identified in our
prior work.
12
11
An obligation is a definite commitment that creates a legal liability of the government for
the payment of goods and services ordered or received, or a legal duty on the part of the
United States that could mature into a legal liability by virtue of actions on the part of the
other party beyond the control of the United States.
12
GAO-16-396. For the 2016 report, we asked a nongeneralizable sample of advertising
experts and professional associations to identify best practices for assessing the
effectiveness of advertising. We compiled and condensed the identified best practices into
a list that we then validated with those same industry experts. That list included three
categories of commercial best practices for assessing the effectiveness of advertising: (1)
organizational structure, (2) planning, and (3) measurement.
Page 5 GAO-20-93 Army Marketing
In addition, we analyzed data from the Enterprise Marketing Management
(EMM) system—the Army’s marketing system of record—on key
performance indicators for measuring the effectiveness of the Army’s
marketing and advertising activities conducted in fiscal year 2018—the
last complete fiscal year of data at the time of our review. To assess the
reliability of the EMM data, we met with AMRG officials to discuss the
processes for entering, maintaining, and reviewing the quality of the data,
and we conducted electronic testing to identify inconsistencies and
outliers. Based on these steps, we determined that the data from fiscal
year 2018 were not sufficiently reliable for the purposes of identifying key
performance indicators associated with the Army’s marketing and
advertising activities. Later in this report, we describe the actions the
Army is taking to address limitations in the reliability of the EMM data.
For the third objective, we reviewed Army general orders, directives, and
other related guidance and documentation on changes to the
organizational structure of the Army’s marketing and advertising program.
In addition, we reviewed the recommendations from two reviews—one
conducted by ASA(ALT) on AMRG’s business processes and the other
conducted by OPM on AMRG’s organizational design. We requested
information and documentation of the Army’s steps to reorganize the
workforce and organizational structure of the marketing and advertising
program and compared this information to applicable key practices for
agency reform efforts we identified in our prior work.
13
Because of the
timing of the reorganization of the Army’s marketing and advertising
program during our review—which we describe later in this report, we did
not assess the Army’s steps against all of the key practices.
We also conducted three focus groups in May and June 2019 with non-
supervisory personnel from AMRG to gain their perspectives on
leadership and workforce changes. We conducted two of these focus
groups at AMRG Headquarters in Arlington, Virginia, and randomly
selected participants from non-supervisory personnel who had been at
AMRG since September 2017. We chose this date because the
personnel would be able to comment on changes in the leadership and
13
GAO, Government Reorganization: Key Questions to Assess Agency Reform Efforts,
GAO-18-427 (Washington, D.C.: June 13, 2018). To identify these key practices, we
reviewed our prior work and leading practices on organizational transformations;
collaboration; government streamlining and efficiency; fragmentation, overlap, and
duplication; high-risk; and other agency longstanding management challenges. In addition,
we interviewed subject-matter specialists knowledgeable about issues related to
government reform and strategic human capital management.
Page 6 GAO-20-93 Army Marketing
workforce practices since officials reported receiving a draft of the AAA
reports, as well as changes in AMRG leadership that occurred in
November 2017. We conducted the other focus group at an AMRG office
in Fort Knox, Kentucky, with non-supervisory personnel who had been at
AMRG since September 2017. In addition, we interviewed five
supervisory personnel who had been at AMRG since September 2017.
We met with non-supervisors separately from supervisors to encourage
candid discussions about the Army’s marketing and advertising program.
We completed written summaries of each focus group and interview, and
then multiple GAO analysts independently coded the responses to
identify themes across the focus groups and interviews and worked to
ensure agreement on the themes.
We conducted this performance audit from December 2018 to November
2019 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.
The Secretary of the Army has the responsibility to recruit personnel,
subject to the authority, direction, and control of the Secretary of
Defense.
14
The Assistant Secretary of the Army for Manpower and
Reserve Affairs serves as the principal advisor to the Secretary for the
Army’s management of its manpower and personnel and provides overall
governance for marketing, advertising, and research.
15
The Deputy Chief
of Staff of the Army, G-1, is the principal military advisor to the Assistant
Secretary of the Army for Manpower and Reserve Affairs and the Chief of
Staff of the Army for all matters related to manpower across the Army. As
of August 2019, the Deputy Chief of Staff of the Army, G-1, has
responsibility for overseeing the new Office of the Chief Army Enterprise
Marketing (AEMO) once it is fully established, as described later in this
report.
14
10 U.S.C. § 7013.
15
10 U.S.C. § 7016.
Background
Roles and Responsibilities
in the Army’s Marketing
and Advertising Program
Page 7 GAO-20-93 Army Marketing
In addition, multiple other Army organizations from across the accessions
enterprise—the collection of Army organizations involved in efforts to
recruit and train soldiers for the Army—have roles and responsibilities in
carrying out the Army’s marketing and advertising program, as
summarized in table 1.
Table 1: Army Organizations with Roles and Responsibilities in the Army’s Marketing and Advertising Program
Army Organization
Roles and Responsibilities
Army Marketing and Research Group
(AMRG), recently redesignated as the
Office of the Chief Army Enterprise
Marketing
a
•
Plan and execute the Army’s national marketing strategy
• Oversee the Army’s primary marketing and advertising contract
• Provide guidance and oversight for the Army’s local marketing and advertising
activities
• Conduct marketing and accessions research
U.S. Army Training and Doctrine Command
(TRADOC)
•
Exercise responsibility for the Army’s accessions mission and recruit and train
soldiers
• Provide guidance and resources specific to supporting the accessions mission
and generating leads
• Develop current market insights and local market innovations
U.S. Army Recruiting Command
(USAREC)
b
•
Recruit Active Duty and Army Reserve soldiers and some officer candidates
• Conduct local marketing and advertising activities in support of its recruiting
goals
U.S. Army Cadet Command
(USACC)
b
•
Administer the Army Reserve Officers’ Training Corps program and, through that
program, recruit Army officer candidates
• Conduct local marketing and advertising activities in support of its recruiting
goals
Office of the Chief, Army Reserve
•
Collaborate with AMRG on national marketing and advertising for the Army
Reserve
• Provide guidance to USAREC and USACC on local marketing and advertising
activities
Army National Guard
•
Support state-level marketing and advertising activities
Source: GAO analysis of Department of the Army information. | GAO-20-93
Note: Information is generally from Army General Order 2019-18, Redesignation, Reassignment, and
Relocation of the U.S. Army Marketing and Research Group as the Office of the Chief Army
Enterprise Marketing, A Field Operating Agency of the Deputy Chief of Staff, G-1 (May 30, 2019);
Army Directive 2018-17, Army Accessions Way Ahead (Oct. 1, 2018); Army Directive 2019-25,
Establishment of the Office of the Chief Army Enterprise Marketing (Aug. 1, 2019); Army Regulation
601-208, The Army Brand and Marketing Program (July 16, 2013); U.S. Army Training and Doctrine
Command Regulation 10-5, U.S. Army Training and Doctrine Command (Apr. 21, 2017); National
Guard Regulation 601-1, Army National Guard Strength Maintenance Program (Jan. 1, 2019).
a
Effective August 2019, the Secretary of the Army redesignated AMRG as the Office of the Chief
Army Enterprise Marketing and reassigned the office to the Deputy Chief of Staff of the Army, G-1.
b
USAREC and USACC are subordinate commands of TRADOC.
Page 8 GAO-20-93 Army Marketing
The Army contracts with a primary advertising agency to develop and
implement its marketing and advertising program. The advertising agency
is responsible for providing a range of services from the development of
the Army’s marketing and advertising strategy to the production of
marketing and advertising activities, including television and print
advertisements, event marketing, and social media. In November 2018,
the Army awarded a contract to a new advertising agency for up to 10
years with a value not to exceed $4 billion.
16
The contract with the
previous advertising agency was awarded in March 2011, and from
March 2011 through the end of fiscal year 2018, the Army issued 702
task orders and obligated about $1.6 billion on this contract, according to
our analysis of data from the Federal Procurement Data System-Next
Generation.
The Army conducts a variety of marketing and advertising activities at the
national and local levels in support of the Army’s recruiting goals.
17
Figure
1 shows examples of the various types of Army marketing and advertising
activities, such as mobile assets used at recruiting events and digital
advertising on social media.
16
The Army issued the original solicitation in January 2017 and awarded the contract in
November 2018. Five proposals were received. The contract award was delayed by
multiple bid protests during the procurement process.
17
The Army distinguishes between national marketing and advertising activities and local
marketing and advertising activities. Until recently, AMRG was responsible for national
marketing and advertising activities, such as the production of broad-reach television
commercials that are coordinated by the Army’s primary advertising agency. AEMO will
carry out these responsibilities going forward, as described later in this report. USAREC,
USACC, and the various state Army National Guard units conduct local marketing and
advertising activities, such as purchasing advertisements in local media outlets.
Types of Marketing and
Advertising Activities
Page 9 GAO-20-93 Army Marketing
Figure 1: Examples of Army Marketing and Advertising
Note: Left image: posting on social media; middle-top image: mobile trailer; middle-bottom:
GoArmy.com website; right image: mobile trailer with video-game consoles.
Different marketing and advertising activities are useful for supporting the
three phases—awareness, engagement, and activation—of an
individual’s decision-making process, sometimes referred to as the
consumer journey. The goal of marketing and advertising is to move a
potential recruit through each of the three phases and, ultimately, to the
decision to enlist:
• Awareness: Individuals learn about the opportunity to serve in the
Army and the distinct characteristics of serving in the Army. The Army
pursues awareness through marketing and advertising activities such
as television commercials, print advertisements, banners at events,
and billboards.
• Engagement: Individuals who are aware of the opportunities for
service in the Army begin considering the possibility of joining the
Army. During this phase, the Army seeks to provide recruits with
additional information to aid in their decision-making process. Often
this phase of advertising takes place in the digital environment, as the
Army seeks to provide informative social media posts and use banner
advertisements to attract individuals to visit its website for more
information.
• Activation: Individuals have considered the Army and are ready to
talk to a recruiter about enlistment. As such, activation activities seek
to encourage these individuals to provide their contact information to
schedule an opportunity to meet with a recruiter. Activation is often
Page 10 GAO-20-93 Army Marketing
conducted in person, such as through recruiters’ presence at events
like career fairs. Activation may also be conducted through other
means—such as direct mail and online or print classified
advertisements—as long as the advertisement prompts viewers to
provide their contact information. Further, the Army employs the use
of mobile assets, such as large trucks and trailers fitted with
equipment and activities intended to draw crowds and encourage and
facilitate public interaction with a recruiter at an event to generate
leads.
Figure 2 shows the awareness, engagement, and activation phases and
examples of marketing and advertising activities that are used in support
of each phase.
Figure 2: Army Marketing and Advertising Phases and Activities in Support of
Recruitment Goals
Page 11 GAO-20-93 Army Marketing
To implement AAA’s recommendations, the Army has developed
processes designed to improve its oversight of the primary contract for
executing the Army’s marketing and advertising program. In April 2018,
AAA reported that AMRG did not sufficiently evaluate the performance of
its primary contractor, effectively oversee deliverables included in its three
main marketing and advertising contracts, or effectively oversee the
negotiation process of task orders for its primary marketing and
advertising contract.
18
AAA made seven recommendations to AMRG to
improve its contract oversight, with which AMRG agreed. As shown in
table 2, as of September 2019, AAA considered three of the
recommendations implemented but not closed, with AMRG still taking
steps to address the other four recommendations; as a result, it is too
soon to assess the extent to which the Army’s steps have improved
contract oversight within the marketing and advertising program.
19
AMRG
officials stated that the implementation of these recommendations has
been slowed, in part, because of the recent award of its primary
marketing and advertising contract to the new advertising agency.
18
At the time of AAA’s review, AAA reported that AMRG had three main marketing and
advertising contracts in place: (1) a primary contract for conducting the Army’s marketing
and advertising program; (2) a contract for marketing and event planning support services;
and (3) a contract for creative technology support. In its report, AAA mainly discussed the
primary contract for conducting the Army’s marketing and advertising program.
19
AAA officials stated that they consider a recommendation as implemented but not
closed when AAA determines that sufficient actions have been taken by the audited
agency, but AAA has not verified that the recommendation’s desired results have been
achieved. Once AAA has verified that the actions have resulted in the desired results,
AAA will classify a recommendation as closed. AAA officials stated that they may conduct
a follow-up audit in fiscal year 2020 to determine if AMRG’s actions are sufficient to close
the recommendations. Appendix II identifies all seven recommendations from AAA’s
report on contract oversight, the actions AMRG has taken to implement them, and the
status of their implementation, according to AAA, as of September 2019.
The Army Has Taken
Steps to Develop
Processes to Improve
Its Oversight of Its
Primary Marketing
and Advertising
Contract
Page 12 GAO-20-93 Army Marketing
Table 2: Status of the Army’s Implementation of U.S. Army Audit Agency (AAA) Recommendations to Improve Contract
Oversight within the Marketing and Advertising Program
Summary of AAA recommendation
Status of implementation
reported by AAA, as of
September 2019
a
Require that all program managers receive the proper contracting officer’s representative training
and provide appointment letters for each one. Ensure quality assurance surveillance plans are
tailored to the task order’s circumstances and there is an individual quality assurance surveillance
plan for all task orders over $150,000.
Not implemented
Update the award fee plan and award fee review process to include soliciting feedback from
program managers, maintaining supporting documentation, and obtaining objective performance
data.
Not implemented
Develop and implement internal procedures to ensure contracts or task orders do not contain
deliverables already provided for in existing firm-fixed-price contracts or task orders.
Not implemented
Use the creative technology support contract for all of the creative technical services within the
scope of its performance work statement.
Not implemented
Use appropriate contract remedies to enforce contract provisions requiring contractors to enter
information into the Contractor Manpower Reporting Application system.
Implemented but not closed
Define and implement a well-structured policy for conducting technical evaluations, to include labor
hours, of cost proposals that presents information in a concise and meaningful manner.
Implemented but not closed
Establish a standardized form to ensure consistency during the evaluation process.
Implemented but not closed
Source: AAA. | GAO-20-93
Note: In April 2018, AAA issued its report on contract oversight within the Army’s marketing and
advertising program. AAA, The Army’s Marketing and Advertising Program: Contract Oversight, Audit
Report A-2018-0033-MTH (Fort Belvoir, Va.: Apr. 13, 2018).
a
AAA officials stated that they consider a recommendation as implemented but not closed when AAA
determines that sufficient actions have been taken by the audited agency, but AAA has not verified
that the recommendation’s desired results have been achieved. Once AAA has verified that the
actions have resulted in the desired results, AAA will classify a recommendation as closed. AAA
officials stated that they may conduct a follow-up audit in fiscal year 2020 to determine if AMRG’s
actions are sufficient to close the recommendations.
Among AMRG’s actions to improve its contract oversight are steps to
develop processes for overseeing contractor performance, deliverables,
and price negotiations. Specifically, AMRG has taken steps in the
following areas:
Created a new project management office. AMRG established a
project management office to help address the challenges identified by
AAA and to serve as a coordinating body that centralizes contract
oversight. According to its charter, the office is responsible for maintaining
cost, schedule, and performance for Army marketing and advertising
programs to help ensure that they are completed on time and within
reasonable costs to support the accessions mission.
Page 13 GAO-20-93 Army Marketing
Documents outlining the contract management process indicate that the
project management office reviews key contract documents, such as the
statement of objectives and the quality assurance surveillance plan,
before the documents are submitted to the contracting officer who is
responsible for administering the contract. After the contractor submits its
proposal for providing the requested product or service, the project
management office reviews the proposal to ensure it meets the
requirements of the statement of objectives. Also, the project
management office coordinates and submits the technical evaluation form
for review by the contracting officer’s representative and the contracting
officer.
Within the newly-formed AEMO, there will be a project management
office with six authorized personnel, according to the organizational chart
for AEMO. In July 2019, AMRG officials stated that they were developing
standing operating procedures and continuity plans that AEMO could use
as it establishes its project management office.
Implementing training and tools to evaluate contractor performance.
AAA found that AMRG did not sufficiently evaluate the performance of its
primary contractor and recommended that AMRG require that all program
managers receive contracting officer’s representative training and ensure
that individual quality assurance surveillance plans are developed for
each task order over $150,000.
20
As of September 2019, AAA reported
that this recommendation had not been fully implemented. In response to
AAA’s recommendations, in April 2019, AMRG reported that all program
managers had completed contracting officer’s representative training.
21
Further, as of September 2019, AMRG reported that the Army had
developed individual quality assurance surveillance plans for each task
order issued in fiscal year 2019. According to AAA officials, AMRG
provided them with examples of these quality assurance surveillance
plans, and AAA provided AMRG with feedback on additional information
20
A quality assurance surveillance plan is required for all service contracts that exceed the
simplified acquisition threshold of $150,000.
21
In September 2019, we found that DOD did not implement an initiative to identify non-
acquisition personnel contributing to services acquisition more generally and did not
comprehensively identify non-acquisition personnel’s need for acquisition-related training.
We recommended, and DOD concurred, that DOD designate an official responsible for
identifying non-acquisition personnel contributing to services acquisitions and ensure
components provide comprehensive information about non-acquisition personnel training
needs. GAO, Defense Workforce: Steps Needed to Identify Acquisition Training Needs for
Non-Acquisition Personnel, GAO-19-556 (Washington, D.C.: Sept. 5, 2019).
Page 14 GAO-20-93 Army Marketing
that should be included, which AMRG officials stated they were taking
actions to address.
In September 2019, AMRG also issued standing operating procedures for
program managers to provide internal policy and instruction for executing
the Army’s marketing and advertising program as well as evaluating its
performance, including overseeing contractor performance. We found that
the standing operating procedures require contracting officer’s
representative training for program managers and that the Director of
Marketing, Director of Research, and AMRG contracts team are to
monitor compliance quarterly. In addition, the procedures include steps
outlining contract oversight mechanisms, such as information on the
purpose and contents of quality assurance surveillance plans.
Implementing a standardized technical evaluation form. AAA found
that AMRG had not effectively negotiated prices for its primary marketing
and advertising contract; AAA recommended that AMRG (1) define and
implement a well-structured policy for conducting technical evaluations of
contractor proposals and (2) establish a standardized form to ensure
consistency during the evaluation process.
22
As of September 2019, AAA
reported that these two recommendations were implemented but not
closed. The standing operating procedures that we reviewed outline the
process for program managers to complete a standardized form for
evaluating contractor proposals. The form is intended to ensure that
program managers are consistently evaluating contractor proposals for
performing work under the contract. For example, the form requires
program managers to conduct a comparative price analysis by comparing
the contractor’s proposed price to total task order cost in prior years.
According to the instructions on the form, program managers are to
submit the completed form to their supervisor or director for approval. The
project management office then coordinates and submits the technical
22
AAA also found that the contractor had not reported labor hours and costs for each task
order in the Contractor Manpower Reporting Application and recommended that AMRG
enforce contract provisions requiring the contractor to enter such information into the
system. As of September 2019, AAA reported that this recommendation was implemented
but not closed. AMRG reported to AAA that the contractor had inputted the required data
prior to the issuance of AAA’s report. In addition, as of April 2019, AMRG officials told us
that the contracting officer’s representative verified that the contractor had inputted the
required information for fiscal year 2017 and was in the process of validating the
information for fiscal year 2018. AMRG also reported to us that the contracting officer will
monitor compliance annually going forward.
Page 15 GAO-20-93 Army Marketing
evaluation to the contracting officer’s representative and contracting
officer for review.
Overseeing contract deliverables. AAA found that AMRG did not
effectively oversee deliverables in its marketing and advertising contracts
and recommended that AMRG develop procedures to ensure that
contracts or task orders do not contain deliverables already provided in
other contracts.
23
As of September 2019, AAA reported that this
recommendation had not been fully implemented. AMRG and U.S. Army
Mission Installation and Contracting Command officials have
implemented processes to prevent duplicative deliverables (i.e., services
or products to be provided through a contract) in the future, such as the
standardized technical evaluation form noted above. We found that the
standardized technical evaluation form requires program managers to
certify that they have reviewed other tasks and contracts within their
purview and to validate that the task order being requested does not
duplicate existing or other requested work. In addition, AMRG and U.S.
Army Mission Installation and Contracting Command officials stated that
they consolidated all contract actions under one team at the U.S. Army
Mission Installation and Contracting Command and that both AMRG’s
contracting officer’s representative and budget office must verify that
contract requests are not duplicative.
AAA also found overlapping deliverables between AMRG’s primary
marketing and advertising contract and a contract for creative technology
support and recommended that AMRG use the creative technology
support contract for all of the creative technical services within its scope.
As of September 2019, AAA reported that this recommendation had not
been fully implemented. AMRG officials told us that they plan to issue a
modification to the contract for creative technology support to remove
services duplicated in the primary marketing and advertising contract. As
of September 2019, an AMRG official told us that the Army expected to
issue the modification in November 2019.
23
AAA also reported that AMRG used a contract for marketing and event support services
to help plan and execute marketing events. However, AAA found that these services were
already provided under the primary contract for conducting the Army’s marketing and
advertising program. In a separate, time-sensitive report, AAA recommended that AMRG
not renew the support services contract or issue an additional task order under the
primary contract. AAA reported that AMRG did not renew the support services contract.
AAA, Time-Sensitive Report: Audit of the Army’s Marketing and Advertising Program –
Support Services Contract, A-2016-MTH-0171.001 (Fort Belvoir, Va.: Aug. 18, 2017).
Page 16 GAO-20-93 Army Marketing
Revising process for contract award fees. AAA found that AMRG had
minimal support to justify its higher award fee ratings for its primary
contractor and recommended that AMRG update its award fee plan and
award fee review process to include soliciting feedback from program
managers, maintaining supporting documentation, and obtaining objective
performance data, among other things.
24
As of September 2019, AAA
reported that this recommendation had not been fully implemented. We
found that AMRG revised its process in fiscal year 2018 for determining
award fee incentives for its advertising agency. For example, according to
the documentation associated with the award fee decision for the
agency’s performance from April 2017 through April 2018, AMRG
reported that it, among other things, included feedback from program
managers on the advertising agency’s performance and obtained
objective performance data from an independent entity, DOD’s Joint
Advertising, Market Research & Studies.
25
In future work with the new advertising agency, AMRG officials stated that
the Army plans to offer award fees for specific task orders rather than one
fee for performance in a given year. According to AMRG officials, this
change allows greater flexibility in deciding which programs should be
incentivized with an award fee. As of September 2019, AMRG officials
stated that they had not issued any task orders with an award fee under
the new marketing and advertising contract.
24
Award fee contracts typically involve fees paid based on contractor performance in
areas that are more subjectively evaluated. Government officials, through an award fee
evaluation board, conduct periodic evaluations of the contractor’s performance against
specified criteria and make recommendations on the amount of fee to be paid. Criteria are
specified in an award fee plan. Following the award fee evaluation, a fee-determining
official makes the final decision about the amount of fee paid to the contractor.
25
Joint Advertising, Market Research & Studies is an official DOD program responsible for
joint marketing communications and market research and studies. This program is to
conduct research about the perceptions, beliefs, and attitudes of American youth as they
relate to joining the military.
Page 17 GAO-20-93 Army Marketing
To implement AAA’s recommendations, the Army has taken steps to
improve how it measures the effectiveness of its marketing and
advertising program; these steps are consistent with commercial best
practices for assessing the effectiveness of advertising identified in our
prior work. In its 2018 report on return on investment, AAA found that
AMRG had deficiencies in how it measured the effectiveness of its
marketing and advertising efforts and made seven recommendations to
AMRG, with which AMRG agreed. Of these recommendations, AAA
considered four implemented but not closed as of September 2019, with
AMRG still taking steps to address the other three recommendations, as
shown in table 3.
26
Since the Army’s steps were recently implemented or
are ongoing, it is too early to determine if they will achieve their desired
results.
26
Appendix II contains all seven recommendations, the Army’s steps to address the
recommendations, and the status of their implementation, according to AAA, as of
September 2019.
The Army Has Taken
Steps to Improve Its
Approach for
Measuring the
Effectiveness of Its
Marketing and
Advertising Program
Page 18 GAO-20-93 Army Marketing
Table 3: Status of the Army’s Implementation of U.S. Army Audit Agency (AAA) Recommendations to Improve the Return on
Investment of the Marketing and Advertising Program
Summary of AAA recommendation
Status of implementation
reported by AAA as of
September 2019
a
Develop specific goals to measure the long-term effects of investments in marketing efforts to support
the Army’s accessions missions.
Implemented but not closed
Revise the Army’s marketing performance framework so that it integrates and leverages marketing
and advertising efforts at both national and local levels.
Implemented but not closed
Establish and formally document a periodic process with roles and responsibilities to assess the
effectiveness and efficiency of all Army marketing and advertising efforts.
Implemented but not closed
Assess the cost-effectiveness of current marketing and advertising efforts and discontinue efforts that
are not cost-effective in comparison to other options.
Implemented but not closed
Establish and formally document a process with roles and responsibilities to ensure that all Army
marketing and advertising performance and cost data is completely and accurately recorded in an
official marketing system of record on a regular basis.
Not implemented
Request update to Army Regulation 601-208 to institutionalize new and revised goals and processes.
Not implemented
Develop a resource requirements projection model that links to and supports planned marketing
efforts.
Not implemented
Source: AAA. | GAO-20-93
Note: In April 2018, AAA issued its report on the return on investment of the Army’s marketing and
advertising program. AAA, The Army’s Marketing and Advertising Program: Return on Investment,
Audit Report A-2018-0036-MTH (Fort Belvoir, Va.: Apr. 13, 2018).
a
AAA officials stated that they consider a recommendation as implemented but not closed when AAA
determines that sufficient actions have been taken by the audited agency, but AAA has not verified
that the recommendation’s desired results have been achieved. Once AAA has verified that the
actions have resulted in the desired results, AAA will classify a recommendation as closed. AAA
officials stated that they may conduct a follow-up audit in fiscal year 2020 to determine if AMRG’s
actions are sufficient to close the recommendations.
Based on our analysis of the Army’s actions, the Army’s steps to
implement AAA’s recommendations fall into the following five areas: (1)
revising strategic goals, (2) updating and documenting its assessment
process, (3) improving the reliability and capabilities of data systems, (4)
integrating national and local marketing and advertising efforts, and (5)
obtaining new tools to determine required marketing and advertising
resources. The steps the Army has taken in these areas thus far are
consistent with commercial best practices for assessing the effectiveness
of advertising we identified in our prior work.
27
As the Army takes
additional steps to establish the newly-formed AEMO, it will be important
27
GAO-16-396. In our prior work, we identified three categories of commercial best
practices for assessing the effectiveness of advertising: (1) organizational structure, (2)
planning, and (3) measurement.
Page 19 GAO-20-93 Army Marketing
for the Army to continue to align its efforts with these commercial best
practices for assessing the effectiveness of advertising to ensure
advertising dollars are used efficiently to help meet stated recruiting
goals.
Revising strategic goals. AAA found that AMRG did not have specific
goals to measure the long-term effects of investments in marketing and
advertising efforts to support the Army’s accessions mission and
recommended that AMRG develop such goals. As of September 2019,
AAA reported that this recommendation was implemented but not closed.
AMRG has revised its strategic marketing goals from tracking changes in
individuals’ attitudes toward the Army, such as support for the Army
among the general population, to tracking the behaviors of these
individuals, such as the number of visits to GoArmy.com. For fiscal year
2018, AMRG had seven strategic marketing goals that tracked the
attitudes of the general population and prospects toward the Army. For
fiscal year 2019, AMRG revised the goals to four that track attitudes, two
that track behaviors, one that tracks effectiveness, and one that tracks
efficiency.
28
Goals that track attitudes are aligned with the awareness
phase of the consumer journey, whereas goals that track behaviors are
more aligned with the engagement and activation phases. Figure 3 shows
how the fiscal year 2018 and 2019 strategic marketing goals align with
the three phases of the consumer journey.
28
AMRG’s effectiveness goal was to increase the conversion rate of leads worked to
appointments made, and its efficiency goal was to decrease the cost of activations on
GoArmy.com. An AMRG official told us that the Army will still have measures of efficiency
and effectiveness for fiscal year 2020.
Page 20 GAO-20-93 Army Marketing
Figure 3: Comparison of the Fiscal Year 2018 and Fiscal Year 2019 Army Marketing Goals to Marketing and Advertising
Phases
Looking ahead, AMRG officials told us that, consistent with feedback they
received from marketing industry experts, the strategic goals in the fiscal
year 2020 marketing plan will all be behavioral and will target the different
stages of what AMRG refers to as a lead nurturing funnel.
29
AMRG
officials stated that their goal is to use information so they can quickly
shift attention and funding to different stages of the funnel that are not
meeting their goals, so as to ensure that those stages get the attention
needed to reach mission success. AMRG’s recent and ongoing steps to
revise its marketing goals are consistent with the commercial best
practice to develop an evaluation framework that identifies the target
audience and includes measurable goals.
To institutionalize AMRG’s updated processes, AAA recommended that
AMRG update Army Regulation 601-208 to reflect the new goals and
processes it would implement to improve its program effectiveness.
30
As
of September 2019, AAA reported that this recommendation had not been
fully implemented. The Army is in the process of revising its marketing
and advertising regulation to reflect the updated strategic marketing goals
and process. AMRG had drafted a revision to the regulation; however,
AMRG officials had put this process on hold while senior Army leaders
were making the decision about AMRG’s placement within the Army. Now
that AMRG has been redesignated as AEMO and reassigned within the
29
The lead nurturing funnel has multiple stages that track the progression from the number
of qualified individuals to the number of individuals who join the Army. The stages include
generating awareness, obtaining contact information, and meeting with potential recruits.
30
Army Regulation 601-208, The Army Brand and Marketing Program (July 16, 2013).
Page 21 GAO-20-93 Army Marketing
Army, as of August 2019 AMRG officials stated that the Army was
revising the regulation to reflect the new organization’s relationship to
other entities within the accessions enterprise. AMRG expects the Army
to publish the updated regulation in 2020.
Updating and documenting its assessment process. AAA found that
only three of AMRG’s 23 national events during fiscal year 2016 provided
the best value for their intended purpose and recommended that AMRG
discontinue efforts that were not cost-effective in comparison to other
options and assess the cost-effectiveness of current marketing and
advertising efforts. As of September 2019, AAA reported that this
recommendation was implemented but not closed. According to AMRG
officials, AMRG discontinued all of the events that were deemed to not be
cost-effective in AAA’s report.
In its report, AAA also found that AMRG’s assessment process did not
include USAREC’s and USACC’s marketing and advertising efforts and
that AMRG did not formally document that process; AAA recommended
that AMRG establish and formally document a process with roles and
responsibilities to assess the effectiveness and efficiency of all Army
marketing and advertising efforts.
31
As of September 2019, AAA reported
that this recommendation was implemented but not closed.
We found that AMRG formally documented how it assesses the
effectiveness of the marketing and advertising program, consistent with
the commercial best practice of seeking to develop an understanding of
how outcomes can be attributed to advertising. In January 2019, AMRG
issued guidance that outlines its assessment process, including the types
of information that are reviewed in each assessment. The guidance we
reviewed outlines three levels of assessments: Level I is a review by
program managers of their individual programs; level II, to be conducted
on a quarterly basis, is a review at the operational level in which the
directors of research and marketing review the results of marketing and
advertising efforts across multiple marketing channels; and level III, also
to be conducted on a quarterly basis, is a review at the strategic level in
31
The recommendation stated that the assessment process should (1) include personnel
from the Assistant Secretary of the Army for Manpower and Reserve Affairs and Deputy
Chief of Staff, G-1; (2) consider cost-effectiveness and contribution in comparison to other
options; (3) include the efforts of USAREC and USACC during the assessment process;
and (4) document decisions to continue, discontinue, or modify efforts to provide best
value in support of the Army’s accessions missions.
Page 22 GAO-20-93 Army Marketing
which the AMRG Director reviews the Army’s progress in meeting its
strategic marketing goals. Further, in January 2019, AMRG reviewed a
summary of the number, cost, and performance of USAREC’s and
USACC’s local marketing and advertising activities as part of the level II
assessment process.
In addition, to facilitate comparison of options, AMRG developed a tool for
comparing different programs within a given marketing channel. The tool,
which AMRG calls a decision support matrix, allows officials to
comparatively rank different programs based on weighting different
factors. For example, based on our review of the decision support matrix,
we found that AMRG assigned a higher weight to a program’s
effectiveness than its timing. The tool also incorporates qualitative
feedback based on how program managers, USAREC, and USACC rank
the programs, and quantitative analysis on the cost per lead, impression,
or engagement, depending on the type of program.
Improving the reliability and capabilities of data systems. AAA
identified discrepancies between information in the Enterprise Marketing
Management (EMM) system and supporting documentation and
recommended that AMRG establish and formally document a process to
ensure that all Army marketing and advertising performance and cost
data were regularly recorded in an official marketing system of record on
a regular basis.
32
As of September 2019, AAA reported that this
recommendation had not been fully implemented. The Army has taken
steps to improve the reliability of the data in EMM since AAA’s report. In
August 2019, the Army issued a task order on its primary marketing and
advertising contract covering EMM system support to include overseeing
and improving the quality of data in EMM. According to the performance
32
In our analysis of EMM data from fiscal year 2018, we also identified potential reliability
issues. For example, our analysis showed that the Army planned for almost 9 million
prospects engaged in person from local marketing and advertising activities, but less than
600,000 actual engagements were reported in EMM. AMRG officials stated that the large
number of planned engagements recorded indicated to them that the data were not
entered correctly, and that they were aware of the issue and were planning steps to
address it, such as issuing a task order for data quality management on the new
marketing and advertising contract. Further, our analysis showed that, of the 836 national
marketing and advertising activities that were identified in EMM as completed before
December 30, 2018, only 15 percent were reported as closed as of June 2019. According
to AMRG officials, for an activity to be considered closed, it should have all of the relevant
cost and performance data entered in EMM. AMRG officials stated that this was possibly
because the Army’s previous advertising agency was often delayed in inputting data and
closing out activities in EMM.
Page 23 GAO-20-93 Army Marketing
work statement, the Army’s new advertising agency is responsible for,
among other things, accurately documenting current data, systems, and
business processes, as well as analyzing EMM reports and
documentation for completeness and accuracy. Further, the advertising
agency is responsible for identifying and documenting business problems
and recommending areas for improvement and technology solutions.
During the focus groups we held with AMRG personnel, participants told
us that AMRG leadership was focused on demonstrating the
effectiveness of the Army’s marketing and advertising through reliable
and readily-available data. For example, the Army implemented an
electronic business reply card, which is a digital form to capture a
potential recruit’s eligibility and contact information and a means of
identifying the event where the recruit learned about the Army.
33
USAREC and USACC officials told us that prior to the electronic business
reply card, recruiters collected prospects’ information at events by using a
paper card. Although that card reflected the event where the potential
recruit heard about the Army, it would often take several days before the
potential recruit’s information appeared in the recruiting system,
according to these officials. As a result, recruiters would sometimes not
send in the paper card and would instead enter the prospect’s information
directly into the system. In these cases, the marketing and advertising
event would not receive credit for generating the lead. USAREC and
USACC officials stated that the electronic business reply card’s quicker
turnaround time for leads showing up in the system should improve data
reliability by ensuring that recruiters and recruiting operations officers
consistently enter an individual’s demographic data into the system, along
with the marketing and advertising activity they interacted with. These
steps to better identify the number of leads generated by marketing and
advertising activities are consistent with the commercial best practice of
conducting ongoing analysis of performance using industry standard
measures appropriate for the purpose of the advertising activity.
In addition, in August 2019, the Army issued a task order on its new
marketing and advertising contract for the maintenance and optimization
of its system that tracks analytics on the Army’s marketing and
advertising activities, which AMRG refers to as the Intelligence Hub. The
advertising agency is responsible for monitoring this system and
33
The electronic business reply card tracks the marketing and advertising activity at which
a potential recruit gave his or her information and helps ensure the digital capture of the
recruit’s demographic information.
Page 24 GAO-20-93 Army Marketing
producing reports that track the effectiveness of marketing and
advertising activities based on key performance indicators. The
advertising agency is also responsible for upgrading the system to track
the multiple marketing and advertising resources that a potential recruit
interacts with. AMRG officials told us that the upgrade of this system is
intended to equip the Army with more complete data to demonstrate the
effectiveness of the Army’s marketing and advertising activities—
consistent with the commercial best practice of seeking to develop an
understanding of how outcomes can be attributed to advertising.
34
Integrating national and local marketing and advertising efforts. AAA
found that AMRG did not integrate and leverage both national and local
marketing and advertising efforts to support the Army’s accessions
mission and recommended that AMRG revise the Army’s marketing
performance framework to include marketing and advertising efforts at
both the national and local levels. As of September 2019, AAA reported
that this recommendation was implemented but not closed. We found that
the Army has created programs and instituted procedures designed to
increase coordination of national and local marketing and advertising
efforts. For example, AMRG reported that it included other Army
components, including USAREC and USACC, in developing the fiscal
year 2019 marketing goals and planned to include those organizations in
its fiscal year 2020 process.
In addition, TRADOC established the Army Accessions Resource Fusion
Board, which brings together organizations from across the accessions
enterprise for quarterly meetings at which they make operational resource
sharing plans for marketing and recruiting assets.
35
For example,
according to a March 2019 briefing for an Army Accessions Resource
Fusion Board meeting, representatives from USAREC brigades
34
AMRG officials stated that the updated system should capture each time an individual
interacts with a marketing and advertising resource, also known as multi-touch attribution.
This capability should allow officials to see all of the marketing and advertising resources
an individual has interacted with, such as digital advertisements clicked on or recruiting
events attended, and to more directly attribute outcomes to specific marketing and
advertising activities.
35
According to its charter, the Army Accessions Resource Fusion Board has a governance
process that includes the (1) Council of Colonels and (2) General Officer Steering
Committee. The Council of Colonels is to meet at least quarterly to review, discuss, and
address items that affect the accessions enterprise. The General Officer Steering
Committee is to meet quarterly as needed to review, discuss, and consider recommended
actions from the Council of Colonels.
Page 25 GAO-20-93 Army Marketing
discussed their planned marketing and advertising activities for the first
quarter of fiscal year 2020, including any requests they had for support
from other Army stakeholders for those planned activities. Further,
according to its charter, the Army Accessions Resource Fusion Board is
responsible for assessing the effectiveness of local marketing and
advertising efforts in the previous quarter.
In fiscal year 2018, the Army also created a pilot program designed to
improve how the Army’s marketing and advertising program coordinates
with its recruiting components and to produce marketing and advertising
messages that resonate more effectively with target populations. The
Army began implementing the program in Chicago in fiscal year 2019 and
as of April 2019 was planning to expand the program to Boston and four
other cities. As of July 2019, AMRG had observed positive results from
the program in Chicago. For example, AMRG reported an increase of 11
percent in the number of leads and an increase of about 7 percent in the
number of recruits who signed contracts with the Army when compared to
the prior year in that region.
Officials from TRADOC, USAREC, and USACC told us that coordination
with AMRG on marketing and advertising efforts has improved since the
time AAA conducted its audits. For example, TRADOC officials stated
that AMRG senior leaders have supported the accessions enterprise by
providing analytic support to USAREC and USACC. Further, USAREC
officials stated that in fiscal year 2018 AMRG started to provide funding
for local marketing and advertising activities near USAREC’s requested
levels, and that this change had been carried forward into fiscal year
2019.
Obtaining new tools to determine required marketing and
advertising resources. AAA found that AMRG did not use a resource
requirements projection model that supported and linked to planned
marketing efforts and recommended that AMRG develop such a model.
36
As of September 2019, AAA reported that this recommendation had not
been fully implemented. AMRG has contracted with the RAND
Corporation and a consulting firm to develop tools to determine the
resources AMRG needs to conduct its marketing and advertising
activities. The RAND tools include three planned models, one of which is
36
A resource requirements projection model predicts the mix of recruiting resources
required for the Army to efficiently meet its accessions mission.
Page 26 GAO-20-93 Army Marketing
the recruiting resource model recommended by AAA. The recruiting
resource model has been partially completed and is being updated with
additional data with full completion scheduled for September 2020.
According to the Army’s fiscal year 2020 budget request, the Army used
the RAND report that developed this model as a justification for
increasing its advertising budget for fiscal year 2020.
Further, in consultation with a consulting firm, AMRG developed a
channel allocation simulator that allows AMRG officials to test different
funding levels for its marketing and advertising channels to see potential
outcomes. For example, based on our review of the simulator, AMRG can
enter a specific amount of funding for events to estimate how many leads
it can expect to obtain from that level of funding. AMRG officials stated
that they can use this tool to help them plan for their required level of
resources for the upcoming fiscal year. The development of this simulator
is consistent with the commercial best practice of using sophisticated
marketing mix modeling to determine an appropriate spending strategy.
37
ASA(ALT) and OPM conducted reviews of AMRG’s workforce and made
recommendations to improve the workforce practices within the marketing
and advertising program. From January to May 2018, ASA(ALT)
conducted a review of AMRG’s business processes and found high-risk
issues that contributed to organizational inefficiencies within five areas:
(1) internal communications, (2) business performance, (3) training, (4)
37
According to industry experts, “marketing mix modeling” refers to the use of statistical
analysis to estimate the effects of various marketing tactics on desired behavior (such as
enlistment) and project the ideal or most efficient mix of tactics to result in that behavior.
The Army Has Taken
Steps to Improve Its
Workforce Practices
and to Reorganize
the Organizational
Structure of Its
Marketing and
Advertising Program
ASA(ALT) and OPM
Conducted Reviews of
AMRG’s Workforce and
Recommended Areas for
Improvement
Page 27 GAO-20-93 Army Marketing
program performance and accountability, and (5) personnel. For example,
ASA(ALT) found that AMRG personnel were unclear about AMRG’s core
mission, objectives, and program priorities. Further, ASA(ALT) found that
AMRG’s personnel, skills, training, and physical locations were not
aligned to support AMRG’s mission. In addition, OPM conducted an
assessment from March to September 2018 to identify organizational
inefficiencies and propose solutions intended to transform AMRG into a
high-performing organization and improve its workforce morale. Similar to
ASA(ALT), OPM identified issues with a lack of mission clarity and
insufficient communication and collaboration throughout AMRG’s
workforce and with its stakeholders. In addition, OPM identified a number
of organizational design issues within AMRG, including workforce
acquisition, management, and optimization of its operational components
and staff.
ASA(ALT) and OPM made multiple recommendations to address these
issues within AMRG. For example, ASA(ALT) recommended that AMRG
establish and disseminate standard operating procedures and process
charts; clarify roles and responsibilities of the various organizational
components; and clearly communicate to staff the final annual marketing
strategy.
38
OPM recommended a multiphased approach to implementing
its overall recommendations, identifying key actions to take in each
phase. For example, within the first phase, OPM recommended that
AMRG determine the new functional structure for AMRG because it would
improve management and accountability, collaboration, and stakeholder
satisfaction. In addition, within the second phase, OPM recommended
that AMRG develop a human capital management plan and review and
update its position descriptions regularly to ensure they align with
changing goals, staffing needs, and the organizational structure of
AMRG.
AMRG took some steps to address ASA(ALT)’s and OPM’s
recommendations to improve its workforce practices. Within its report,
ASA(ALT) noted that AMRG had started to take actions to implement
several recommendations, such as disseminating AMRG’s mission
statement, priority objectives, strategic goals, and fiscal year 2019 annual
marketing plan guidance to all AMRG personnel. Similarly, OPM noted
38
In addition to its recommendations related to the organization and workforce practices of
AMRG, ASA(ALT) recommended taking steps to improve AMRG’s business performance
and budget planning and execution processes to increase spending oversight, among
other things.
Page 28 GAO-20-93 Army Marketing
that AMRG had established task groups to coordinate with stakeholders,
participated in meetings with Congress and stakeholders, and was
developing a new vision for AMRG. However, as of April 2019, senior
AMRG officials stated that they had not taken steps to address all of the
reports’ recommendations because the Army was considering broader
organizational changes to the placement of AMRG within the Army. While
the recommendations in ASA(ALT) and the OPM reports were generally
specific to AMRG’s organization and workforce at that time, senior AMRG
officials stated that the Army would take additional steps to incorporate
ASA(ALT)’s and OPM’s recommendations, as appropriate, after senior
Army leadership made decisions about those changes.
In our review, AMRG personnel we met with continued to identify poor
internal communications and morale as key challenges within AMRG,
consistent with the findings from ASA(ALT) and OPM. During our focus
groups with AMRG personnel, participants repeatedly stated that senior
AMRG leadership did not communicate key information to staff. For
example, participants told us that senior AMRG leadership did not
communicate information about AMRG’s mission, strategic priorities, or
pending organizational changes. As described below, subsequent to our
focus groups, the Army began taking steps to fundamentally change the
organizational structure, workforce, and leadership of its marketing and
advertising program. In light of the timing of these substantial changes,
we did not comprehensively assess the extent to which communication
issues have been resolved in the reorganization of the marketing and
advertising program. It will be important for the new leadership to focus
on communication at the outset of this organizational change to establish
positive morale within the workforce.
Page 29 GAO-20-93 Army Marketing
The Army recently reorganized its marketing and advertising program to
improve its organizational structure; the Army’s early steps to implement
the reorganization are consistent with some key practices for agency
reform efforts we identified in our prior work, as described below.
39
In May
2019, the Secretary of the Army reassigned, redesignated, and stated
that the Army planned to relocate AMRG.
40
The Secretary of the Army
redesignated AMRG as AEMO and reassigned the office to the Deputy
Chief of Staff of the Army, G-1. The effective date of this reassignment
was August 1, 2019. AEMO’s mission is to coordinate the Army’s national
marketing and advertising strategy; develop and maintain relationships
with the marketing and advertising industry; and develop marketing
expertise and talent within the Army to support the Army, Army National
Guard, and Army Reserve accessions. The offices will be moved from
Arlington, Virginia, and Fort Knox, Kentucky, to Chicago, Illinois.
Consistent with the key practice to designate a leader to be responsible
for the implementation of the proposed reforms, the Secretary of the Army
designated the Assistant Secretary of the Army for Manpower and
Reserve Affairs as responsible for establishing AEMO and overseeing the
transition. The Assistant Secretary stated that he expected AEMO to be
fully operational by early 2020.
The Army identified several reasons for transitioning from AMRG to
AEMO and reassigning the office to the Deputy Chief of Staff of the Army,
G-1, consistent with the key practice to define and articulate a succinct
and compelling reason for the reforms. According to the execution order
establishing AEMO, the Army needed an organization strategically
positioned to: support Army senior leadership in advertising, marketing,
and analysis; coordinate with the Army’s primary advertising agency; be
talent diverse; provide effective marketing analysis; and be able to
provide consistency of message and brand across the Army accessions
enterprise. The Assistant Secretary of the Army for Manpower and
39
GAO-18-427. Agencies should consider these key practices in the development and
implementation of agency reforms that include any organizational changes, such as
transformations and other reorganizations, and efforts to streamline and improve the
efficiency and effectiveness of government operations. Categories of key practices to
assess agency reform efforts include (1) goals and outcomes; (2) processes for
developing the reforms; (3) implementing the reforms; and (4) strategically managing the
federal workforce. Each of these four broad categories has relevant subcategories and
key practices.
40
Army General Order 2019-18, Redesignation, Reassignment, and Relocation of the U.S.
Army Marketing and Research Group as the Office of the Chief Army Enterprise
Marketing, a Field Operating Agency of the Deputy Chief of Staff, G-1 (May 30, 2019).
The Army Reorganized Its
Marketing and Advertising
Program to Improve Its
Structure, and Early Steps
Are Consistent with Some
Key Reform Practices
Page 30 GAO-20-93 Army Marketing
Reserve Affairs also stated that AEMO is being assigned to the Deputy
Chief of Staff of the Army, G-1, in part, because of the continuity in
leadership that having a military officer lead the organization will provide.
Previously, AMRG was assigned to the Office of the Assistant Secretary
of the Army for Manpower and Reserve Affairs, whose leader is politically
appointed and had been vacant for two years until January 2019. In
addition, the Assistant Secretary stated that AEMO will be located in
Chicago to increase its coordination with the new advertising agency,
which is also headquartered in Chicago. The Assistant Secretary also told
us that the Army hoped to recruit civilian staff and to leverage the
marketing and advertising expertise at academic and other marketing and
advertising institutions in the region.
The Army has taken some initial steps to establish AEMO and its
operations. The Army has outlined a three-phased plan with specific tasks
and associated dates within each of these phases, which is consistent
with the key practice to establish implementation goals and a timeline to
build momentum and show progress for the reforms. Phase 1, which was
to be completed by August 1, 2019, prioritized tasks to initially establish
AEMO, such as publishing the Army directive establishing AEMO, issuing
the execution order outlining roles and responsibilities for the transition
from AMRG to AEMO, and identifying office space in Chicago. Phase 2,
which is to be completed between August 1, 2019, and February 1, 2020,
includes tasks to transition AEMO to being fully operational, such as
establishing new position descriptions and equipping the permanent office
space. Lastly, phase 3 identifies those tasks to be implemented after
February 1, 2020, when AEMO is fully operational and conducting daily
operations, such as updating roles and responsibilities in the Army’s
regulation for its marketing and advertising program and developing
policies to direct commission military personnel to the office.
41
The plan also identifies offices and officials who are accountable for
implementing specific tasks during the transition. The Assistant Secretary
of the Army for Manpower and Reserve Affairs told us that the Army
established an operational planning team to execute the transition from
AMRG to AEMO. The execution order also identifies key stakeholders,
including officials from TRADOC, Office of the Chief, Army Reserve, and
41
The Army allows recruits with certain backgrounds, such as medical, legal, or cyber, to
gain a direct commission as an Army officer without attending an Army Reserve Officers’
Training Corps program, Officer Candidate School, or the U.S. Military Academy at West
Point.
Page 31 GAO-20-93 Army Marketing
National Guard Bureau, who are to participate in weekly working group
meetings led by the Assistant Secretary of the Army for Manpower and
Reserve Affairs. Looking forward, by January 2020 the Army plans to
develop metrics to assess the effectiveness of the new AEMO
organization, including the purpose, expectations, and desired outcomes,
which is consistent with the key practice that calls for clear outcome-
oriented goals and performance measures.
In addition, the Army has taken some initial steps to establish AEMO’s
workforce. As of September 2019, the Army had authorized 53
positions—31 military and 22 civilian—for AEMO and identified a
Brigadier General from the Army Reserve with marketing and advertising
experience as its leader. Senior Army leadership stated that they
expected to fill almost all of the positions with new military and civilian
personnel, in part, because the civilian position classifications in AMRG
do not generally align with those in AEMO. As of June 2019, the Assistant
Secretary of the Army for Manpower and Reserve Affairs told us that the
Army was identifying Active Duty, Reserve, and National Guard officers
with marketing and advertising education or experience to fill the military
positions. In addition, the Assistant Secretary stated that they were
working with OPM to develop position descriptions for the AEMO civilian
personnel and to identify the skills and expertise needed within AEMO to
fulfill its mission, consistent with the key practice to determine if the
agency will have the needed resources and capacity, including the skills
and competencies, in place for the reorganization. The Army has also
established a working group led by the U.S. Army Office of Economic and
Manpower Analysis at West Point to develop a new marketing career
path that senior Army leadership stated is intended to create a pool of
military personnel who could serve in AEMO and other Army accessions
organizations in the future.
As the Army carries out its steps to fully establish AEMO and reorganize
the marketing and advertising program, it will continue to be important for
the Army to consider and use the key practices for agency reform efforts
to guide the transition. Doing so will help ensure the success of the new
marketing and advertising organization.
We provided a draft of this report to DOD for review and comment. The
Army provided technical comments, which we incorporated as
appropriate.
Agency Comments
Page 32 GAO-20-93 Army Marketing
We are sending copies of this report to the appropriate congressional
committees, the Secretary of Defense, and the Secretary of the Army. In
addition, the report is available at no charge on the GAO website at
http://www.gao.gov.
If you or your staff have any questions about this report, please contact
me at (202) 512-2775 or f[email protected]. Contact points for our Offices
of Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made key contributions to this report
are listed in appendix III.
Elizabeth Field
Director, Defense Capabilities and Management
Appendix I: Army Actions to Improve Its
Marketing and Advertising Program as Stated
in the Army’s Report to Congress
Page 33 GAO-20-93 Army Marketing
This appendix summarizes an Army report to Congress on actions taken
to improve its marketing and advertising program.
1
Section 599 of the
John S. McCain National Defense Authorization Act for Fiscal Year 2019
required the Army to submit a report to Congress that addressed several
elements, such as the mitigation and oversight measures implemented to
assure improved program return and contract management, and the
establishment of a review process to regularly evaluate the effectiveness
and efficiency of marketing efforts.
2
The Army submitted the report on
February 5, 2019. Table 4 identifies the required elements of the report
and the actions that the Army has reported taking to address these
elements.
Table 4: The Army’s Reported Actions to Congress on Improving the Army’s Marketing and Advertising Program
Element
Summary of Army’s Reported Actions
The mitigation and oversight measures
implemented to assure improved program return
and contract management including the
establishment of specific goals to measure long-
term effects of investments in marketing efforts.
In the report to Congress, the Army stated that the Army Marketing and
Research Group (AMRG) took several steps to address this element, such as
revising its fiscal year 2019 performance goals and performance framework to
enhance short-term and long-term performance, integrating national and local
marketing and advertising efforts through a pilot program, and implementing
processes for contractor oversight.
Since issuing the report, the Army has continued to take steps to address this
element with plans to revise its fiscal year 2020 marketing goals and expand
its pilot program into additional cities. AMRG also established a project
management office to integrate and standardize contract oversight.
The establishment of a review process to regularly
evaluate the effectiveness and efficiency of
marketing efforts including efforts to better support
the accessions missions of the Army.
In the report to Congress, the Army stated that AMRG cancelled the 20
national marketing and advertising events that the U.S. Army Audit Agency
(AAA) reported were not cost-effective, used a decision support matrix to
evaluate and prioritize annual marketing programs, contracted with the RAND
Corporation to quantify the cost-effectiveness for marketing and advertising
activities, and established data analysis tools to better track the effectiveness
of its programs.
Since issuing the report, the Army has continued to take steps to address this
element by redesigning its main Army website to tailor content to individual
visitors and issuing a task order on its new marketing and advertising contract
for the maintenance and optimization of its system that tracks analytics on the
Army’s marketing and advertising activities.
1
Department of the Army, Army Marketing and Advertising Program: Army Audit Agency
Recommendations on Contract Oversight and Return on Investment: Report to Congress
(February 2019).
2
Pub. L. No. 115-232, § 599 (2018).
Appendix I: Army Actions to Improve Its
Marketing and Advertising Program as
Stated in the Army’s Report to Congress
Appendix I: Army Actions to Improve Its
Marketing and Advertising Program as Stated
in the Army’s Report to Congress
Page 34 GAO-20-93 Army Marketing
Element
Summary of Army’s Reported Actions
The increase of acquisition and marketing
experience within AMRG.
In the report to Congress, the Army stated that AMRG was transitioning
marketing specialist positions to specialized marketing and program
management skills and providing program managers with contracting officer’s
representative training.
Since issuing the report, the Army has implemented a project management
office to integrate and standardize contract oversight, which will be included in
the new Office of the Chief Army Enterprise Marketing. In addition, the Army is
creating a marketing career field for military personnel who are assigned to
the new office.
A workforce analysis of AMRG in cooperation with
the U.S. Office of Personnel Management (OPM)
and industry experts assessing the AMRG
organizational structure, staffing, and training,
including an assessment of the workplace climate
and culture internal to the AMRG.
In the report to Congress, the Army stated that OPM in October 2018
submitted its final report with recommendations and that the Army had
contracted with industry leaders to provide input on the organizational design
and job functions within AMRG.
Since issuing the report, the Army redesignated AMRG as the Office of the
Chief Army Enterprise Marketing and reassigned it from the Assistant
Secretary of the Army for Manpower and Reserve Affairs to the Deputy Chief
of Staff of the Army, G-1. To establish the new office, the Army has developed
a new organizational structure and, in consultation with OPM, is developing
position descriptions for personnel.
The establishment of an Army Marketing and
Advisory Board comprised of senior Army and
marketing and advertising leaders and an
assessment of industry and service marketing and
advertising best practices, including a plan to
incorporate relevant practices.
In the report to Congress, the Army stated that AMRG would routinely engage
with industry leaders and the other services to identify industry best practices.
The Army stated that it took this approach, instead of an Army Marketing and
Advisory Board, because it was consistent with the statute’s intent and
avoided restrictions from the Federal Advisory Committee Act, which governs
the establishment and operation of advisory committees.
Since issuing the report, the Army reported that AMRG has continued to
engage with industry leaders through a consulting contract, such as obtaining
feedback on industry best practices on how to revise its strategic marketing
goals and framework.
The status of the implementation of contracting
practices recommended by the AAA audit of
contracting oversight of the AMRG.
In the report to Congress, the Army stated that AMRG was regularly reviewing
its progress in implementing AAA’s recommendations.
Since issuing the report, the Army has continued to monitor its progress in
implementing the recommendations. AAA officials stated that they have
worked closely with AMRG to monitor the status of implementing the
recommendations and that AMRG has sought feedback on steps taken to
implement the recommendations.
Source: GAO analysis of Department of the Army information. | GAO-20-93
Note: Section 599 of the John. S. McCain National Defense Authorization Act for Fiscal Year 2019
required the Army to submit a report to Congress on the Army’s steps to improve the contract
oversight and return on investment of its marketing and advertising program, among other things.
Pub. L. No. 115-232, § 599 (2018).
Appendix II: Army Marketing and Research
Group’s Reported Actions to Address Army
Audit Agency’s Recommendations
Page 35 GAO-20-93 Army Marketing
This appendix provides a summary of actions that the Army has reported
taking to address the recommendations in the April 2018 U.S. Army Audit
Agency (AAA) reports about the Army’s marketing and advertising
program.
1
One report focused on contract oversight, and the other report
focused on what AAA termed “return on investment.” Each report
contained seven recommendations, with which the Army Marketing and
Research Group (AMRG) concurred. AMRG and AAA officials stated that
they have communicated about AMRG’s actions to implement the
recommendations and that AAA has provided feedback, as appropriate,
on actions taken. AAA officials stated that AAA may conduct a follow-up
audit in fiscal year 2020 to determine if the actions have led to
improvements in the marketing and advertising program. Table 5
summarizes the recommendations from the AAA report on contract
oversight, the actions AMRG has taken to implement them, and the
status—as of September 2019—of their implementation as reported by
AAA.
1
AAA, The Army’s Marketing and Advertising Program: Return on Investment, Audit
Report A-2018-0036-MTH (Fort Belvoir, Va.: Apr. 13, 2018) and The Army’s Marketing
and Advertising Program: Contract Oversight, Audit Report A-2018-0033-MTH (Fort
Belvoir, Va.: Apr. 13, 2018).
Appendix II: Army Marketing and Research
Group’s Reported Actions to Address Army
Audit Agency’s Recommendations
Appendix II: Army Marketing and Research
Group’s Reported Actions to Address Army
Audit Agency’s Recommendations
Page 36 GAO-20-93 Army Marketing
Table 5: Army Marketing and Research Group’s (AMRG) Actions to Address U.S. Army Audit Agency (AAA)
Recommendations on Contract Oversight
Summary of AAA recommendation
AMRG’s actions
Status of
implementation
reported by AAA,
as of September
2019
a
Require that all program managers
receive the proper contracting
officer’s representative training and
provide appointment letters for
each one. Ensure quality
assurance surveillance plans are
tailored to the task order’s
circumstances and there is an
individual quality assurance
surveillance plan for all task orders
over $150,000.
As of April 2019, AMRG program managers had received contracting
officer’s representative training. AMRG planned to issue appointment
letters according to the requirements of individual task orders.
AMRG also developed quality assurance surveillance plans for each
task order issued in fiscal year 2019 under the previous marketing and
advertising contract as well as for the 16 task orders issued under the
new contract as of September 2019.
In September 2019, AMRG developed standing operating procedures
for marketing and procurement activities, which state that program
managers are required to obtain contracting officer’s representative
training and that the Director of Marketing, Director of Research, and
AMRG contracts team are to monitor program managers’ certification
quarterly.
Not implemented
Update the award fee plan and
award fee review process to
include soliciting feedback from
program managers, maintaining
supporting documentation, and
obtaining objective performance
data.
For the award fee period of April 7, 2017, to April 6, 2018, AMRG
included program manager feedback, surveyed Award Fee Review
Board members who were familiar with the agency’s performance,
used supporting documentation, and considered objective
performance data from Joint Advertising, Market Research & Studies.
Under the new primary marketing and advertising contract, AMRG
plans to update the award fee process so that the award fee option will
be available at the task-order level, which AMRG officials stated will
provide greater flexibility to choose which programs should be
incentivized by an award fee.
Not implemented
Develop and implement internal
procedures to ensure contracts or
task orders do not contain
deliverables already provided for in
existing firm-fixed-price contracts
or task orders.
AMRG implemented several updates to the contracting process to
ensure there are no duplicative contracts, such as consolidating all
AMRG contract actions under one team at the U.S. Army Mission
Installation and Contracting Command and setting up a multilayered
approval process to verify that contract requests are not duplicative
with existing contracts.
AMRG has also developed a standardized technical evaluation form
that requires program managers to certify that the task order being
requested does not duplicate existing or other pending work.
Not implemented
Use the creative technology
support contract for all of the
creative technical services within
the scope of its performance work
statement.
AMRG plans to issue a modification to the contract for creative
technology support to remove services duplicated in the primary
contract for conducting marketing and advertising activities. As of
September 2019, AMRG expected to issue the modification in
November 2019.
Not implemented
Use appropriate contract remedies
to enforce contract provisions
requiring contractors to enter
information into the Contractor
Manpower Reporting Application
system.
In November 2017, AMRG validated that 2017 contractor manpower
data was input as required into the Contractor Manpower Reporting
Application system. As of April 2019, AMRG was continuing to validate
that the contractor had entered the required data for fiscal year 2018.
The contracting officer will also validate contractor-input data on an
annual basis.
Implemented but not
closed
Appendix II: Army Marketing and Research
Group’s Reported Actions to Address Army
Audit Agency’s Recommendations
Page 37 GAO-20-93 Army Marketing
Summary of AAA recommendation
AMRG’s actions
Status of
implementation
reported by AAA,
as of September
2019
a
Define and implement a well-
structured policy for conducting
technical evaluations, to include
labor hours, of cost proposals that
presents information in a concise
and meaningful manner.
In September 2019, AMRG issued standing operating procedures for
program managers that provide guidance on developing task order
requests and evaluating proposed task orders by completing a
technical evaluation form.
Implemented but not
closed
Establish a standardized form to
ensure consistency during the
evaluation process.
AMRG has developed a standardized form to conduct technical
evaluations prior to task order award. The form requires program
managers to perform a comparative price analysis with the requested
activity and prior year’s costs; confirmation that requested travel is
necessary; project purpose and AMRG coordination; project schedule;
and review of subcontractor bids.
Implemented but not
closed
Source: GAO analysis of Department of the Army information. | GAO-20-93
Note: In April 2018, AAA issued its report on contract oversight within the Army’s marketing and
advertising program. AAA, The Army’s Marketing and Advertising Program: Contract Oversight, Audit
Report A-2018-0033-MTH (Fort Belvoir, Va.: Apr. 13, 2018).
a
AAA officials stated that they consider a recommendation as implemented but not closed when AAA
determines that sufficient actions have been taken by the audited agency, but AAA has not verified
that the recommendation’s desired results have been achieved. Once AAA has verified that the
actions have resulted in the desired results, AAA will classify a recommendation as closed. AAA
officials stated that they may conduct a follow-up audit in fiscal year 2020 to determine if AMRG’s
actions are sufficient to close the recommendations.
Appendix II: Army Marketing and Research
Group’s Reported Actions to Address Army
Audit Agency’s Recommendations
Page 38 GAO-20-93 Army Marketing
Table 6 identifies the recommendations from the AAA report on return on
investment, the actions AMRG has taken to implement them, and the
status—as of September 2019—of their implementation as reported by
AAA.
Table 6: Army Marketing and Research Group’s (AMRG) Actions to Address U.S. Army Audit Agency (AAA)
Recommendations on Return on Investment
Summary of AAA recommendation
AMRG’s actions
Status of
implementation
reported by AAA as
of September 2019
a
Develop specific goals to measure
the long-term effects of
investments in marketing efforts to
support the Army’s accessions
missions.
AMRG revised the marketing goals for fiscal year 2019 to shift from
attitudinal goals to behavioral goals, including four goals that reflect
perceptions of the Army, two behavioral goals, one efficiency goal, and
one effectiveness goal. For fiscal year 2020, AMRG expects to shift
entirely to behavioral goals based on the stages of a lead nurturing
funnel, which AMRG officials stated will allow AMRG to evaluate its
effectiveness at different stages and to more quickly allocate money to
different stages as needed to ensure mission success.
Implemented but not
closed
Revise the Army’s marketing
performance framework so that it
integrates and leverages marketing
and advertising efforts at both
national and local levels.
AMRG integrated the U.S. Army Recruiting Command (USAREC),
U.S. Army Cadet Command (USACC), Army National Guard, Office of
the Chief, Army Reserve, and the Office of the Chief of Public Affairs in
the fiscal year 2019 marketing planning process. According to AMRG
officials, the results of USAREC’s and USACC’s local marketing and
advertising efforts are incorporated into one of the fiscal year 2019
strategic marketing goals—the conversion rate of the number of leads
to appointments made.
The Army also created a pilot program designed, among other things,
to redefine how the Army’s marketing and advertising program
coordinates with its recruiting components and to produce messaging
that resonates more effectively with target populations. The first pilot
was in Chicago and will be expanded to Boston and four other cities.
Implemented but not
closed
Establish and formally document a
periodic process with roles and
responsibilities to assess the
effectiveness and efficiency of all
Army marketing and advertising
efforts.
In January 2019, AMRG issued guidance that outlines its assessment
process, including the types of information that are reviewed in each
assessment.
AMRG also developed a decision support matrix, a tool that values
and ranks national marketing and advertising programs. According to
AMRG officials, this tool ranks the programs by weighting factors
including qualitative feedback from USAREC, USACC, and the Office
of the Chief, Army Reserve, and quantitative metrics such as cost per
impression or lead.
Implemented but not
closed
Assess the cost-effectiveness of
current marketing and advertising
efforts and discontinue efforts that
are not cost-effective in
comparison to other options.
AMRG discontinued the 20 marketing and advertising events that AAA
found were not cost effective.
Implemented but not
closed
Appendix II: Army Marketing and Research
Group’s Reported Actions to Address Army
Audit Agency’s Recommendations
Page 39 GAO-20-93 Army Marketing
Summary of AAA recommendation
AMRG’s actions
Status of
implementation
reported by AAA as
of September 2019
a
Establish and formally document a
process with roles and
responsibilities to ensure that all
Army marketing and advertising
performance and cost data is
completely and accurately
recorded in an official marketing
system of record on a regular
basis.
In August 2019, the Army issued a new task order on its primary
marketing and advertising contract for EMM system support to include
overseeing and improving the quality of data in EMM. According to the
performance work statement, the Army’s new advertising agency is
responsible for, among other things, accurately documenting current
data, systems, and business processes, as well as analyzing EMM
reports and documentation for completeness and accuracy. Further,
the advertising agency is responsible for identifying and documenting
business problems and recommending areas for improvement and
technology solutions.
As of April 2019, AMRG reported that it was transferring all of its
vendors’ customer data into EMM. Going forward, the vendors and the
advertising agency will enter performance and cost data directly into
EMM. AMRG expected to have the new application for performance
and cost data developed and implemented by October 2019. AMRG
officials told us that once that process is complete, AMRG would
document the process and associated roles and responsibilities for
entering information into EMM.
Not implemented
Request update to Army
Regulation 601-208 to
institutionalize new and revised
goals and processes.
AMRG developed a draft version of the updated Army Regulation 601-
208, which had been informally staffed across all affected
organizations including the Deputy Chief of Staff of the Army, G-1,
TRADOC, USAREC, and USACC. As of July 2019, AMRG had not yet
finalized the regulation because of recent organizational changes in
the marketing and advertising program, according to AMRG officials.
AMRG officials told us they expect the Army to publish the revised
regulation in 2020.
Not implemented
Develop a resource requirements
projection model that links to and
supports planned marketing efforts.
The Army contracted with the RAND Corporation to develop a model
to determine the expected resource needs to achieve the annual
recruiting goals for the Active Army and Army Reserve, including the
number of recruiters, amount of incentives, and marketing and
advertising funding. The RAND Corporation is scheduled to complete
this model by September 2020. In addition, the Army contracted with
the RAND Corporation to develop a model to address AMRG’s future
funding needs. The RAND Corporation is expected to complete this
model by February 2021.
AMRG also developed a marketing channel allocation simulator tool
that allows decision makers to see the effect of different funding levels
for marketing channels on outcomes, such as impressions or leads, at
different stages of the lead nurturing funnel.
Not implemented
Source: GAO analysis of Department of the Army information. | GAO-20-93
Note: In April 2018, AAA issued its report on the return on investment of the Army’s marketing and
advertising program. AAA, The Army’s Marketing and Advertising Program: Return on Investment,
Audit Report A-2018-0036-MTH (Fort Belvoir, Va.: Apr. 13, 2018).
a
AAA officials stated that they consider a recommendation as implemented but not closed when AAA
determines that sufficient actions have been taken by the audited agency, but AAA has not verified
that the recommendation’s desired results have been achieved. Once AAA has verified that the
actions have resulted in the desired results, AAA will classify a recommendation as closed. AAA
officials stated that they may conduct a follow-up audit in fiscal year 2020 to determine if AMRG’s
actions are sufficient to close the recommendations.
Appendix III: GAO Contact and Staff
Acknowledgments
Page 40 GAO-20-93 Army Marketing
In addition to the contact named above, Margaret Best (Assistant
Director), Kendall Banks, Timothy J. DiNapoli, Jacob Fender, Alexandra
Gonzalez, Amie Lesser, Kristen Kociolek, Steven Lozano, Jonathan
Meyer, Eve Nealon, Julia Kennon, Carol Petersen, Richard Powelson,
Jerome Sandau, Jared Sippel, and Andrew Stavisky made key
contributions to this report.
Appendix III: GAO Contact and Staff
Acknowledgments
GAO Contact
Staff
Acknowledgments
(103226)
The Government Accountability Office, the audit, evaluation, and investigative
arm of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of the
federal government for the American people. GAO examines the use of public
funds; evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make informed
oversight, policy, and funding decisions. GAO’s commitment to good government
is reflected in its core values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost is
through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO
posts on its website newly released reports, testimony, and correspondence. To
have GAO e-mail you a list of newly posted products, go to https://www.gao.gov
and select “E-mail Updates.”
The price of each GAO publication reflects GAO’s actual cost of production and
distribution and depends on the number of pages in the publication and whether
the publication is printed in color or black and white. Pricing and ordering
information is posted on GAO’s website, https://www.gao.gov/ordering.htm.
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card, MasterCard,
Visa, check, or money order. Call for additional information.
Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
Visit GAO on the web at https://www.gao.gov.
Contact FraudNet:
Website: https://www.gao.gov/fraudnet/fraudnet.htm
Automated answering system: (800) 424-5454 or (202) 512-7700
Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400,
U.S. Government Accountability Office, 441 G Street NW, Room 7125,
Washington, DC 20548
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548
James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707
U.S. Government Accountability Office, 441 G Street NW, Room 7814,
Washington, DC 20548
GAO’s Mission
Obtaining Copies of
GAO Reports and
Testimony
Order by Phone
Connect with GAO
To Report Fraud,
Waste, and Abuse in
Federal Programs
Congressional
Relations
Public Affairs
Strategic Planning and
External Liaison
Please Print on Recycled Paper.
