Ag2Manip is a framework designed to enable robots to learn novel manipulation skills without relying on expert demonstrations. It introduces agent-agnostic visual and action representations to bridge the gap between human and robot manipulations, enhancing generalization and adaptability. The visual representation is derived from human demonstration videos, with humans and robots obscured to reduce domain-specific biases. The action representation abstracts robot movements into a universal proxy agent, simplifying complex actions and focusing on essential interactions between the end-effector and object. Ag2Manip achieves significant improvements in task success rates, reaching 78.7% in simulated environments, compared to 18.5% for baselines. It also enhances imitation learning success rates from 50% to 77.5% in real-world settings. The framework's effectiveness is validated through extensive simulations and real-world experiments, demonstrating its practical applicability and generalizability across various tasks and environments. Key contributions include an agent-agnostic visual representation that narrows the embodiment gap, an agent-agnostic action representation that simplifies complex actions, and significant progress in robot skill learning performance. Ag2Manip's approach addresses challenges in autonomous skill acquisition, including domain gaps and sparse task executions, by leveraging generalizable representations and a structured reward function. The method's success is further supported by ablation studies and experiments on real-world imitation learning, highlighting its potential for real-world applications in robotic manipulation.Ag2Manip is a framework designed to enable robots to learn novel manipulation skills without relying on expert demonstrations. It introduces agent-agnostic visual and action representations to bridge the gap between human and robot manipulations, enhancing generalization and adaptability. The visual representation is derived from human demonstration videos, with humans and robots obscured to reduce domain-specific biases. The action representation abstracts robot movements into a universal proxy agent, simplifying complex actions and focusing on essential interactions between the end-effector and object. Ag2Manip achieves significant improvements in task success rates, reaching 78.7% in simulated environments, compared to 18.5% for baselines. It also enhances imitation learning success rates from 50% to 77.5% in real-world settings. The framework's effectiveness is validated through extensive simulations and real-world experiments, demonstrating its practical applicability and generalizability across various tasks and environments. Key contributions include an agent-agnostic visual representation that narrows the embodiment gap, an agent-agnostic action representation that simplifies complex actions, and significant progress in robot skill learning performance. Ag2Manip's approach addresses challenges in autonomous skill acquisition, including domain gaps and sparse task executions, by leveraging generalizable representations and a structured reward function. The method's success is further supported by ablation studies and experiments on real-world imitation learning, highlighting its potential for real-world applications in robotic manipulation.